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Item 20 (Continued) 

electronic system. This system was used as the test bed for this model battery. The soldering and general test 
equipment JTPT are applicable to all electronic technicians. The other tests of the battery apply to technicians 
concerned with this specific doppler radar system. 

Each class of activity fur which JTPT v.ere developed contilins its individual mix of behaviors, but it is not 
mutually exclusive. There are dependencies among the classes. As a result a four level hierachy of dependencies can 
be stated: (1) checkout., removal/replacement, and soldering* (2) use of general and special lest equipment; (3) 
alignment/calibration, and (4) troubleshooting. For example, troubleshooting may include all the activities 
mentioned before it. 

Due to the diverse character of the various mixes of behaviors involved in each class of maintenance activity, a 
single score report of test results would be meaningless. A profile of test results tlierefore was developed which 
provides for an individual cell for each test problem. The tests are structured so that each problem produces a 
product. The results for each problem is reported in terms of a go, no-go score. Either the test subject produces a 
satisfactory product or he does not. Where time is important, tie must produce the satisfactory product m a specified 
time. AItlu>ugh process may be valuable as a diagnostic tool, it is not considered as an appropriate factor for scoring 
purposes. 

The hieraehy of dependencies mentioned previously has implication for the order in which tests are 
administered as well as for diagnostics. For example, since troubleshooting includes the use of test equipment and 
other activities in the hiernchy, logic would dictate that administration of the tests for the sub-activitits would 
precede the troubleshoot*ng tests and that a test subject would not be permitted to take the troubleshooting tests 
ntil he had passed these other subtest. 

Due to the unavailability of a sufficient number of experienced test subjects ai the time of ihc tryout of the 
JTPT battery, the tryout was not as extensive as planned. The limited tryout did indicate that the tests as developed 
are administratively feasible.. There continued use, no doubt, would result in further modifications and polish. The 
report also includes a discussion of several implementation considerations and suggestions. 
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SUMMARY 



Problem 

The in-depth review of the literature reported in Volume I of this series of reports strongly reiterated 
the fact that paper and pencil tests of job knowledge and electronic theory tests have very poor 
criterion-related or empirical validity with respect to the ability of electronic maintenance men for 
performing titeir job tasks. This literature review, also produced little evidence that anything was being 
accomplished in a systematic way to develop adequate job measures. Most of the reported test efforts were 
of the "ad hoc" variety. Such tests usually covered only parts of the maintenance job and attempted to 
report results in term^ of a single score, providing little diagnostic information. Another facet of this 
problem was (hat even though criterion-referenced job performance tests were recognized as being superior 
by many training people, paper and pencO tests were substituted because they were more easily and cheaply 
developed and administered. 

Approach 

Hypotheses were suggested for improving the job coverage and scoring of Job Task Performance 
Tests (JTPT) and for developihg symbolic substitute tests that would have higher empirical validity than the 
traditioial paper and pencil job knowledge tests. Such symbolic substitutes could not be developed until 
good JIPT were avaDable as criteria. This effort was aimed a* *'ie development of such criterion tests. The 
hypotheses that were offered for improving JTPT included the following: 

1 Separate tests could be developed for each type of job activity. The activities to be considered 
were checkout procedures; align, adjust, and calibrate activities; remove and replace activities; 
troubleshooting; use of test equipment; and use of hand tools. 

2. A separate and appropriate scoring and diagnostic scheme could be developed for reporting the 
results from each type of test.. 

Based on these hypotheses plans were made to develop a battery of JTPT together with appropriate 
scoring schemes for each type of electronic maintenance activity. The doppler radar, the AN/APN-147 and 
its computer, the AN/ASN-35, were selected as typical equipments for test development. .These plans called 
for a testing system that could be administered by personnel that were not experts in the maintenance of 
these equipments. The system was to include the JTPT, an administrator's handbook, and a training 
program for test administrators. Thi.s system was to be considered as model for the development of future 
tests for measuring ability to perform electronic maintenance tasks.. The model was to be appropriate for 
use in field maintenance units, for use in training, and for use in research projects. The plans called for a 
rather extensive tryout of this testing system in field maintenance units in the Military Airlift Command 
(MAC). 

Results 

A model JTPT system was developed. This system included 48 JTPT that covered the typical 
organizational and intermediate maintenance activities mentioned; and administrator's manual in 
step-by-step format and a training program for Air Force administrators. The training program was given 
successfuDy to members of the MAC maintenance standardization team. Each test was administered 
successfuDy at least seven times to experienced Air Force mairtenance personnel., Due to fewer 
standardization team field visits than planned during the tryout period and to a "non-interference" policy 
for the MAC standardization team, the plans to gather additional h';rd data concerning electronic 
technicians' job abilities were not achieved. 

As to the scoring scheme, product, process and time were considered as to their appropriateness. A 
test subject has not reached criterion on a task until he has produced a complete satisfactory product .,'T\\\% 
is a go, no-go criterion. In most cases, time is not a critical factor but foi'some organizational (fiight line) 
tasks It miglit be. Where time is critical, the subject must produce this product within a prescribed time. 
Otherwise, le must produce his product in a reasonable time. The product for each type of test will of 
course bo st mewhat different. Process information was not considered an appropriate basis for a score but 
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such iiiiurination may be used for diagnostic purposes. During the lest development and tryout it became 
apparent that the activities were not mutually exclusive. These activities can be arraiKed in a ur level 
hierachy of dependencies as follows: 

I . Check out* remove and replace* and soldering activities. 

2 Use of general and special test equipment. 

3. Align, adjust, and calibrate activities. 

4. Troublesliooting. 

This hierarchy of dependencies among activities together with the go, no-go product criterioif for 
ca<.h test problem made possible the development of a diagnostic profile for reporting test results (Figures 
10 and 11). 

In addition to the reporting of the development and tryout of the JTPT system, the report discusses 
other relevant factor> concerning the development and implementation of JTPT. Some examples follow. 
The soldering tests and the tests on use of general equipment have general application from equipment to 
equipment, whereas the other tests are equipment specific. Current personnel and training systems are built 
around paper and pencil testing practice but an orderly modification of training and testing practice is 
possible 

Conclusions 

The tryout of the JTPT system indicated that the system is an adequate model for JTPT on other 
equipments, that Air Force Maintenance Technicians can be successfully trained as test administrators and 
that they are able to administer tests to Air Force Maintenance personnel ..The tests on soldering and use of 
general test equipment together with appropriate training packages can be used in the field electronic 
maintenance shops and in formal and on-the-job training piograms., These actions can be accomplished with 
a minimum of modifications of current Air Force maintenance and training programs. Such action would 
result in great benefit to tlie Air Force. The use of equipment specific tests in the field and in training 
would require a much more expensive test development program and an extensive modification of training 
and maintenance procedures. But until maintenance, training and personnel specialists and administrators 
accept and understand the fact that most paper and pencil job knowledge tests and theory tests have 
extremely low empirical validity, we can expect no large scale shift to JTPT., 



ERLC 



PREFACE 



Tliis document represents a portion of the Exploratory Development program of the Advanced 
Systems Division of the Air Force Human Resources Laboratory. It is a compilation and expansion of 
materials submitted by URS/Matrix Research Company, Falls Church, Virginia 22042 under contracts 
F33615^9C.1232 and F33615-70C-1695. Dr. Edgar L. Shriver was the Principal Investigator. 

This document is Part I of the second volume (AFHRL-TR-74-57(n), Part I) of a four volume report 
to be published concerning the evaluation of maintenance performancc./The other documents are entitled; 

1 Evaluating Maintenance Performance: An Analysis. AFHRL-TR-74-57(l). In press. 

2. Evaluating Maintenance Performance: Test Admmistrator's Manual and Test Subject's 
Instructions for Criterion Referenced Job Task Performance Tests for Electronic Maintenance. AFHRL- 
TR-74-5 7(11), Part II. In press. 

3. Evaluating Maintenance Performance: The Development of Graphic Symbolic Substitutes for Criterion 
Referenced Job Task Performance Tests for Electronic Maintenance. AFHRL-TR-74-57(lll). In press. 

4. Evaluating Maintenance Performance:, A Video Approach to Symbolic Testing of Electronic 
Maintenance Tasks. AFHRL-TR-74-57(IV). In press. 

The preparation of all these documents has been documented under task area and work unit 171010, 
Evaluating the Performance of Air Force Operators and Technicians of Project 1710, Training for Advanced 
Air Force Systems. Identification of this document by work units included work units 17101003, 
17100416 and 17101006. Dr.. John P. Foley was task scientist. Dr. John P. Foley and Mr. John K. Klesch 
of the Advanced Systems Division shared the contract monitorship. Dr. Ross L Morgan was the Project 
Scientist.. 

The authors wish to acknowledge the cooperation and assistance of many individuals who 
contributed to this effort. Mr.. John F. Hayes and Mr. William R. Hufliand of URS/Matrix Research 
Company expended a great deal of effort during the development and tryout of the Job Task Performance 
Test (JTPT) system. At various stages during the planning, development and tryout many Military Airlift 
Command people were involved. Some of the MAC Headquarters personnel that should be mentioned 
include Col Donald H. Watt, Lt Col Shelby Corpsman, Lt Col Lloyd B. McKeethen, Capt Gary E. Qark, 
Capt Warren E, Spenser, Capt Robert A. Letz, Special mention is due to CMS Jack Brown and SMS John 
Beaucher of the MAC Maintenance Standardization Team who served as test administiators. At McGuire 
AFB, Col Geraid Auger, Lt Col Edward Kaiatt, CMS Wilbur Easey, CMS James Townes, Mr., Milton Wills 
and M:.. David Bond. At Norton AFB, Lt Col Leon Creed. At Altus AFB, Maj Louis P. Gerac and Capt 
Stephen R. Millers. Mr., John K. Klesch of the Advanced Systems Division, AFHRL, must be thanked for 
his contributions as well as for his several extended trips to operational sites. Dr. Ross L. Morgan and Dr. 
Gordon A. Eckstrand provided many helpful suggestions. 
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EVAULATING MAINTENANCE PERFORMANCE: THE DEVELOPMENT AND TRYOUT OF 
CRITERION REFERENCED JOB TASK PERFORMANCE TESTS FOR ELECTRONIC MAINTENANCE 



SECTION I. TEST DEVELOPMENT METHODOLOGY 
Introduction 

In AFHRL"TR-74-57(l) of this series of reports, tlie background of evaluating technical personnel was 
described and the case for performance based evaluations made. AFHRbTR-74-57(I) therefore is recom- 
mended as a valuable prelude for a fuller appreciation of this volume. Based on that report, this project was 
undertaken to develop a series of criterion referenced performance tests that achieve valid evaluations of 
readiness, training on-the-job performance, job performance aids, and other new concepts for the 
maintenance process. 

The main thrust of the efforts reported in this volume was for the development of an administratively 
feasible battery of criterion referenced Job Task Performance Tests (JTPT) and adequate scoring schemes 
for the key tasks of electronic maintenance jobs. The types of such key tasks considered are outlined in 
AFHRL-TR-74-S7(I) and include checkout, use of hand-tools, use of general and special test equipment, 
remove/replace, align/adjust/calibrate, and troubleshooting. The hand-tools and general test equipment 
JTPT concern activities which have general use among all electronic technicians. Although the battery was 
developed as a model, the remainder of the tests is systems specific to the doppler radar, the AN/APN-147, 
and its computer, the AN/ASN-35. The battery as developed includes 48 tests. 

During and after development each of these tests was administered at least seven times to insure its 
administrative feasibility. This administrative feasibility requirement was achieved. Snce these tests were 
designed to be as near to the ultimate job criterion as possible, no validation of their empirical validity was 
possible or necessary. The tests are empirically valid by definition. Original plans called for additional field 
administrations of the battery for the purpose of obtaining some hard data on how well technicians can 
perform the key tasks of their jobs. Unfortunately, due to lack of subject time, this desirable secondary 
objective was not achieved. 

JTPT, in the context of this report, refer specifically to evaluating an individual's ability to perform a 
task that has been identified as a component of his present or anticipated job. In this rationale, no 
"knowledges" are separately teste !. They may be applied or not applied. What is tested is the ability to 
perform the task, and the evidence is a completed job product which meets job specifications. The test 
subject may have used knowledges and physical skills to produce that product, but no attempt is made in 
these tests to separate the knowledge and skill components for individual evaluation. Ability to perform the 
job task in such a way as to produce the job task products is accepted as prima facie evidence of possession 
of requisite knowledges. 

In addition to measuring ability to perform job tasks, these performance tests were also to be 
developed so as to provide "diagnostic" information. The test should be able to provide some indicatior\of 
the specific nature of a performance deficiency even though the test does not assign scores to factors 
contributing to over-all performance. 

The training/testing milie.j into which the products of a project such as this one will fit can be 
described as "job-referenced," "performance-oriented," or "criterion referenced " That is, the prime 
emphasis of such a system wi/i be the production of personnnel equipped with the necessary training, 
job-aids, and techniques to pe. nit them to accomplish assigned job tadcs under operational conditions. The 
criterion of performance is production of products which meet job specifications. This is a "go, no-go" 
criterion for the production of each job product. 

Objectives 

Based on the discussion in AFHRLTR-74-57(I) and the preceding discussion, the objective of this 
project was to develop as a model a set of JTPT that serve the purposes specified and have the following 
characteristics. 



Job Relevance or Empirical Validity. Performance is evaluated by inspecting Wit job products of each 
task performed in the test. If tlie job products are those required on the job, then the test & job relevant. 
There is no more ultimate test of test validity than this. 

Scorability. Each test item must be scorable objectively. The score for each item should indicate 
whether the task has been correctly performed or not (go, no-go). The score is given when tlie job product 
of the task performed meets job specifications. If the product does not meet specifications no score is given 
for the test item no matter how much of the performance process was "correct." The criterion of norform- 
ancc is in the product. The score is not given for correctly performing steps in the job proce 
producing a job product according to job specifications. Under certain conditions, **time \ ' 'in idy 
become part of a task **go, no-go" specification. 

Representativeness. A job requires numerous kinds of products, processes or inputs. It is not feasible 
to test for all. But each kind must be tested in order to determine if the test subject is capable in all these 
areas. The test constructor must state which kinds of tasks the test tests. But he cannot prove that there is 
not some other kind of task in the job. He can only state what kinds of tasks the test does test.. 

Process Independence. E^ch test item must measure only whether the job product is produced or not. 
This measure must be independent of the process for producing that product. This means the test can be 
used to evaluate alternative processes; e.g., different documentation or training for the job. 

Diagnostics. Since the test does not measure or give scores for steps in the performance process, it 
provides no formal indication of "why" a test subject failed to produce a particular job product. But the 
test can be designed to provide supplementary information for such diagnostic purposes. Furthermore, 
these diagnostics may be organized into a formal framework of dependent »es to be organized into subtests. 
For instance, a subtest on "specia! tool use" should be administered before personnel are tested on job 
products that require "special tools" for ;beir products. Since production of job products is dependent on 
use of special vOols, no useful purpose is served 'by testing personnel on job products until it is determined 
that they can use the special tocfW. However, this diagnostic framework is based on assumptions made for 
efficiency in testing. The ultimate criterion is always production of job products and subtests can only be 
referenced to that criterion.. 

Administrative Feasibility.. The test situation must be such as to allow the test subject to produce the 
job product or a product that is tantamount to that. This imposes a considerable administrative effort, far 
greater, for instance, than would be required for a paper and pencil test of job knowledges assumed to be 
prerequisites for the job. But to make the administration of a criterion reference test feasible there may be 
need to separate the main test into a series of subtests referenced to the job products. Careful though and 
analyses should go into all the administrative details for maximizing the number of personnel tested in a 
given period of time. 

Standardization of Test Conditions. Test conditions should be standardized except for the treatment 
under test. For instance, if the treatment of subjects being tested is use of different documentation, than all 
other conditions should be standardized; e.g., hand-tools, location of spares or lubricants, test equipment, 
etc. This does not mean that multiple treatments cannot be tested at the same time; e.g., documentation 
and training, but then the results cannot be traced to one treatment or the other, but must be attributed to 
whatever was allowed to vary. 

Scope of Coverage 

The tasks that maintenance men perform for the Air Force are many and varied. A convenient model 
for indicating and classifying this wide variety of activities is presented in Figure 1. Hie various types of 
maintenance activities (such as, checkout procedures; align, adjust and calibrate procedures; trouble- 
shooting; remove and replace procedures; and the use of tools and test equipment) are represented on one 
axis of the model. Since mechanical equipment and electronic equipment usually require a different variety 
of maintenance actions, they are represented by another axis. TTie third axis of the model represents the 
three levels or categories of maintenance now found in the military services. Organizational maintenance is 
the first level. It is usually aimed at checking out a whole machine subsystem and correcting any identified 
faults as quickly as possible. Right line maintenance falls in this category. A system is checked out. If it 
does not work, the line replaceable unit (LRU) or "black box" causing the malfunction is identified and 
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replaced. This major component is then taken to the field shop (intermediate maintenance) where it is again 
checked out and the faults, authorized for correction, are corrected. The corrective actions, authorized at 
the intermediate level, vary greatly from system to system depending on the maintenance concept of each 
system. On some systems, the maintenance man will troubleshoot the "black box" to the piece part level. 
In more modern equipment, he will identify a replaceable module made up of many piece parts. Some 
modules are thrown away, others sent to the depot for repair. Any line replaceable units which the field 
shop are unable, or unauthorized to repair are sent to the depot for overhaul. 

Organizational and intermediate level organizations are manned primarily by enlisted iechnicians 
whose average length of service is rather short (slightly more than 4 years in the Air Force). Depots are 
manned largely by civilian personnel with a much higher level of experience and longer retention time. 
Using this model it has been possible to specify areas of concentration for study. 

The scope of this test devdopment effort includes the shaded cells of the model. The actual tests 
were to be developed for the intermediate level of maintenance but all of thejob activities of the organiza- 
tional level of maintenance are included in the intermediate level. The only difference is that these common 
job activities are usually performed in a different job environment., (For example, for organizational 
maintenance on aircraft these common tasks are performed on the flight line while for intermediate 
maintenance they are performed in the field shop.) 

The AN/APN-147-AN/ASN-35 system was used ^s a test bed for the application of the model. To 
broaden the coverage would be to reduce the practicality of the model for implementation, and restrict it 
to an abstract level of conceptual thinking and discussion. The type of tests described here can be used by 
supervisors or standardization teams to test the job capability of personnel at any level of maintenance. But 
more importantly, the tests can be used to measure the effectiveness of any job proce$ses,job aids, or job 
training, because they are process independent. 

The intention of this effort was to add to the concept of criterion referenced JTPT by specifics 
generated through implementation and practice. The authors do not know how far the specifics can be 
stretched to become generalities. But, a reasonable assumption is that the tests in the battery which has 
been developed can serve as models for tests for measuring the effectiveness of maintenance on other 
electronic hardware. 

Approach 

During the course of this project an approach to criterion referenced JTPT and scoring scheme 
development evolved that appears to be a valid general model, at least for classes of maintenance outlined 
previously. 

Main Elements of Approach 

The main elements for implementing this approach are as follows: 

The first step is to identify the major classes or kinds of tasks that are involved in the job or skill area 
under consideration. For this project, these classes have been initially identified in AFHRL-TR«74-57(I) and 
are discussed above under Scope of Coverage. They include equipment check-out, alignment/adjustment, 
component replacement, fault isolation, test equipment usage, and hand tool usage. These classes served as 
the representative kinds of tasks required by thejob. During the development process and test try-outs, 
certain shifts in emphasis and definition became necessary. 

The second step is to define and describe the input, processing and output conditions for each of the 
kinds of tasks established in Step One. The output requirements were stated in terms of the specifications 
of the output products in each class of tasks. In electronic maintenance the product is always an item of 
equipment repaired and ready for field use. To produce this product the test subject must perform certain 
processes on the inputs to him. Even though the process which the test subject uses is not measured directly 
in the criterion referenced JTPT, it is still the performance process that is of interest to us. For instance, a 
given unit which has been repaired represents one product. But it can fail in many ways, each of which 
requires a certain repair process. After any repair the product is the same, a unit ready for field use. But we 
are interested in the performance process. We measure its accomplishment by the product. But we want to 



know if the test subject can flx the equipment regardless of its input condition. The input conditions 
should be those common to the job but standardized for the test.: 

It is also important to consider the performance processes in order to make diagnoses from the test 
performances, that is, to help answer questions about why the subject does not produce a criterion output 
as the result of his performance process. B^' analyzing and defining the process in general terms, such 
diagnostic categories as ''test equipment** or 'hand tools** can be identified as potential areas of weakness 
in electronic maintenance. That is, a common cause of failure to produce criterion outputs may be 
traceable to an inability to produce the proper outputs on test equipment or from the improper use of hand 
tools. These common causes are themselves candidates for subtests. That is, subtests can be established for 
"use of test instruments'* or other "sub-routines** in the performance process for the whole job. Any 
common causes of difficulty should have tests constructed for them.. And personnel should be able to 
perform correctly on those tests before taking a test for the whole job. 

The third step is to select certain job products or job processesin each kind of job duty for the test.. 
In some jobs it may be possible to test all the job products in one kind of job duty. But that is not always 
administratively feasible, and in the present case of electronic technicians it certainly is not. In the case of 
an electronic technician the "job products'* of troubleshooting are at least as large in number as the number 
of parts in the equipment because a different job process is involved in repairing each part. There is a 
question of "how many** or "what percentage'* of the total number of job processes or outputs will 
constitute a "representative*' sample. There is no absolute answer to this question. 

Whatever answer is arrived at, short of testing 100% of the job products, is subject to challenge. It can 
be argued that one process or output not tested out of a thousand tested would not show the same results 
as the others. Admittedly, objecting to one process out of a thousand is a feeble argument. A job supervisor 
would be happy to get and electronic technician who could solve a thousand problems, but be of doubtful 
ability on only one.: In fact, a job supervisor might be happy to get a man who could solve only half the 
problems in the equipment. This example is given to show that "representativeness** is not jjist a theoretical 
question, but a practical one. In a practical sense, the question may reduce to one of specifying to the job 
supervisor what percentage of the job outputs the test subject has been tested on and let him say whether 
he would be satisfied with a man of proven ability on that job sample. The ultimate question of "represent- 
ativeness** comes down to such practical considerations. B!it there are other actions to be taken to improve 
representativeness in the test. 

In second step the processes of producing job products were examined. This provides the test 
constructor with an idea of further subdivisions within a given kind of job area. For instance, in the 
category of "alignment** several different processes might be found.: One alignment might be identified 
which requires disassembly of many parts and another that requires none.. This indicates- somewhat 
different processes involved in the performance of each. The test should include some products from each 
of these categories. 

In developing the present model test, the adjustment/alignment/calibrate category had 40 job 
products. The troubleshooting category had thousands of job products (one for each part). The equipment 
was subdivided into electronic stages, and one job product selected from each stage to gain represent- 
ativeness.. But representativeness cannot be guaranteed. After taking a battery of criterion referenced JTPT 
we can say quite accurately what tasks the test subject can do, but we can't go beyond that. It is essentially 
up to the user to accept or reject such a person. 

Figure 2 presents a flowchart type representation of the relationship among the kinds of tasks for 
electronic jobs. The solid lines indicate the series of relationships found to be the most common pattern of 
technicians* actions. The dotted lines indicate possible but less likely relationships. 

The fourth step is to construct a framework of subtests referenced'^ to the main criteria. This is done 
for efficiency in testing, not for theoretical reasons. In fact, theoretically it is undesirable. It would be 
desirable to test each person on a full scale of criterion referenced lests, covering every possible task of his 
job, then there would be no question of the relevance of the performance required.: But for practical 
reasons the subtests are introduced to make the administration of the tests feasible. The framework of 
subtests is a rational one, but the reasoning on which it is based is always subject to some question. 
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However, with the criterion reference avaflable, the questions can always be answered. If a representative 
group of men fafl on a subtest but can produce the job products in a fuD scale of criterion referenced tests, 
the reasoning or assumptions on which the subtest are based are incorrect and the subtest must be revised 
or eliminated. So there is a "fail safe" feature to subtests which are referenced to the job criterion. 

The framework is constructed on the basis of assumed dependencies. Table 1 presents a structure for 
considering this rationale of dependencies. This table indicates that troubleshooting is the most complex 
and demanding task. Within the concept of troubleshooting is included not only fault isolation but also the 
necessary activities for returning the equipment to field ready use once a fault is isolated. Within the fault 
isolation portion of troubleshooting, several types of activities are required. Fault isolation usually requires 
the use of a checkout procedure to surface "out of tolerance" conditions. It requires the use of some 
cognitive activity usually structured in terms of some sort of troubleshooting scheme, strategy or logic. But 
the execution of this scheme requires that the troubleshooter gather information about the operation of the 
equipment under test. This requires the proper use of test equipment. It makes little difference how good 
his cognitive scheme for fault isolation is if his improper use of test equipment is feeding him faulty 
information about the equipment's operation. 

If we give a subject a fault to isolate and he fails to isolate it, his lack of ability could be: the result of 
improper checkout, a faulty cognitive process, improper use of test equipment or inability to use hand 
tools. We have no way of knowing which one from this test alone. But if we have given him subtests on use 
of test equipment, on use of hand tools and on the checkout procedure and he has demonstrated that he 
can perform these activities, we can be reasonably sure that his cognitive processes are at fault. If on the 
other hand such subtests indicate that he cannot use his test equipment property, there is little to be gained 
by concentrating on the cognitive processes until his lack of ability to use his test equipment has been 
corrected. 

Once the troubleshooter has isolated the fault he must take the necessary corrective action to return 
the equipment to field ready condition. This could include align and adjust action, or remove, replace or 
repair actions including soldering, A weakness in any one or all of the eight activities indicated in Table 1 
for troubleshooting could result in the troubleshooter not returning the faulty equipment to field ready 
condition. 

The table also indicates that align and adjust procedures, as well as, remove and replace, repair, and 
checkout procedures may under certain Job conditions be considered independent activities, each in its own 
right, with its own set of dependencies. The tests and subtest developed during this effort reflect the 
framework indicated in this table and in Figure 2.. 

The fifth step is to determine the means for measuring the task output or product that will indicate 
that the task has been conipletely and correctly performed. This is accomplished by inspecting the final 
product to see if it meets specifications. In practice the test administrator does not necessarily have to make 
a special measurement of his own. As a p'^cal matter he can observe the unit ch^k-out results obtained 
by the test subject in lieu of making Oftr checks himself. But the test should state, for the Test Admin- 
istrator (TA), what the specifications for each product are (by reference or with a special piece of paper, 
template, or whatever is necessary). The TA merely scores the test subject as "go" if his product meets a// 
specifications or as "no-go" if any product specifications are not met. 

It should be noted that some products become "sealed units*' when they are repaired. In this event 
the TA must be told to make certain inspections before the unit is "sealed." For instance, a lubricant may 
be placed in a unit before it is closed up. The test administrator cannot determine whether the correct 
lubricant has been used, or if any has been used, by inspecting the sealed unit. He must observe the 
markings on the lubricant container during the performance process to see if specifications have been met.. 
There are other instances in which the easiest and best way to measure the product specifications is to 
observe some aspect of the performance process. But this performance is not scored as a performance. It is 
only used as a convenient way of measuring part of a product specification. 

The sixth step is to determine how to simulate the input and output conditions for the test in the 
same way they occur on the job.. In the case of mechanical or electronic maintenance the input conditions 
may occur in several ways.. For instance, a supervisor may be part of the input conditions. He might say, 
"There is a crack in the drive shaft. I want you to take this section out and replace it." Or the same man 
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might discover the same crack while lubricating a joint on the drive shaft. Or it might be discovered during a 
routine inspection. The question for the test developer becomes one of "which condition to simulate." The 
answer is quite straightforward. It is to state clearly what it is you are simulating in the test. If you want to 
test the man's ability to remove and replace the driveshaft, start with input conditions that tell him to 
remove and replace it, just as a supcrvi«tor would do on the job. If you want to test his ability to recognize a 
crack then use input conditions which start with an inspection. The test constructor must state what it is 
that has been tested, and he must give the specifications for it. 

He must also use the proper iftput conditions for the situation he says he is testing. For the test this 
input condition must be standardized; e.g., the same crack for all test subjects. A hairline crack should not 
be used for one man and an eigjith-inch gash for another. The same thing is true for failure.: The cause 
usually must remain a constant condition of the test so long as it is being used for a particular purpose. In 
addition to specifying the means for measuring the main job products, subtests also require the Cfvctioti of 
simulated job products or outputs. For instance, it special test equipment such as an oscilliscope is 
separated out as a subtest, a "simulated product" may have to be generated for it. By making this a subtest, 
we have taken away the regular job product: a repaired item of equipment. We must create a replacement 
product. The replacement should impose the same tiemands on the test subject, with respect to the 
oscilliscopey that the regular job does. The substitute product should have output specifications similar to 
the real job product. ^Iso a similar input should be provided. 

But the procesi can be attenuated. An example of this attenuation is as follows: A substitute 
equipment to be measured by an oscilliscope would be generated. It would produce a signal simUar to the 
real equipment. But it would not require the test subject to find the right place to measure, or even what 
the measurement should be.. It would not require any further processing other than to be measured by an 
oscilliscope. This greatly attenuates the job process and leaves only the requirement for a measurement by 
the oscilliscope. This attenuation is a major way in which the administration of the battery of criterion 
referenced JTPT is simplified and made feasible. 

But this item of substitute equipment must give the test subject no hint of how to operate his 
oscilHscopc to get the right output from the equipment.: The output of this substitute box is known by the 
TA. The test subject's job is to compare his reading with a reading given him fo*- this substitute equipment 
and say whether it is good or bad. This is the same thing the job requires. The test should not require the 
test subject to verbalize or write the output in volts, microseconds, frequency , etc., but only to say good or 
bad, as the job requires. Examples of such substitute products for subtests are presented later in this 
volume . 

The seventh step w development of the administraHve arrangements for the tests^ The decisions 
regarding criterion test items and their specifications have a direct impact on how the test will be admin- 
istered. It must be remembered that the (administered) situation for each test item must allow for the test 
subject to produce a job referenced product or output. Within this condition every effort must be made to 
make the tests simple to administer. In some cases it may be necessary to reexamine the criterion product 
in order to devise a more administratively feasible method of evaluating performance. 

The primary administrative considerations are cost, time, test security, administrator-test subject 
ratio, equipment degradation, and administrative instructions. After establishing the administrative details 
of the test, a final product or output still must be producted or the process performed must be tantamount 
to producing the product. An example of a process stopped short of a product which is considered 
tantamount to production of a pioduct is as follows: 

A faulty part is inserted in the equipment by the test administrator. The test subject's job is to restore 
the equipment to field ready condition. He goes through whatever process he chooses to accomplish this. 
But to achieve the final product the final step in this process is to unsolder the faulty part and solder in a 
good one (then check to see that this accomplishes the Tepaix).' Administratively this last step is undesirable. 
Administratively it is much more efficient to stop the test subject before he unsolders the faulty part. 
(Soldering and unsoldering places much wear and tear on equipment and, in a test situation, equipment 
would soon wear out around this faulty part.) Also the equipment with the same faulty part is needed for 
the "next" test subject.. But if we accede to these administrative efficients, we are not producing the 
product of a field ready item of equipment. In the present tests this dilema was resolved by saying that the 
identification of the faulty part was tantamount to producing the final product. Soldering and unsoldering 
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of parts were tested in a subtest, and it was assumed that if the test subject could solder and unsolder parts 
in a subtest he could do the same in the main test. But this was an assumption. It is open to question at 
any time. It can be answered by having the test subject complete the job product. This option is always 
available in a criterion referenced JTPT. But this does not mean that the option must always be exercised. 
The present writers feel that it is sufficient for the option to exist and the assumption identified as open to 
challenge. This is the same basis on which a framework of subtests is establislied. These concessions are 
made to make the tests more administratively feasible and here the matter must rest. The test constructor 
can resolve the administrative dilemmas any way he chooses. But he must state what he has done so that 
the ultimate criterion reference is always attainable if sufficient administrative resources are expended to 
achieve it. It is not desirable to make these administrative concessions. To some extent they weaken the 
strength of the reference to the job criterion. But it is not a bad compromise so long as the reference thread 
is not lost and can be established to "test the test" at any time. 

Other Approach Considerations 

The following discussion broadens the issue just introduced to aspects of the electronic maintenance 
job other than troubleshooting to a faulty part.: In performance testing, the equipment damage considera- 
tion is a major one for several reasons. First, the equipment must operate properly and reliably to permit 
smooth progress of the test administration. Equipment down time can ruin a testing schedule. Secondly, 
since students will be altering the equipment, there is a possibility of equipment detuning and of test 
compromise by overworking a given part of the equipment.. 

Several steps can be taken to overcome these tendencies. First ^^^P ihe test subject's equipment 
alteration requirements to a minimum. In the troubleshooting portion of the tests developed in this project, 
test subjects are not permitted to unsolder and connections. A replacement routine is utilized instead. The 
test subject soldering capability is then tested on work samples which are not part of the prime equipment. 
When It was not possible to conduct the test without test subject alteration of the equipment, as in 
alignment or component removal and replacement, detailed instructions and aids were developed for the 
Test Administrator, so that he could quickly restore equipment to its proper operating condition without 
the use of sophisticated test equipment. 

Seooncl test administration procedures and instructions should be set up so that the normal technical 
supervisor can administer the tests with a minimum of difficulty. By keeping the administrator require- 
ments for technical expertise to a minimum, the probabflity that the tests will be properly administered will 
be increased, .\dministration instructions must be explicitly detailed and amply supplemented by graphic 
depictions of equipment test configurations. In addition. Test Administrators should be given practice in 
administering he tests to become familiar with testing idiosyncrasies. 

Tryout and Revision 

The eighth and final step in. test development is tryout and revision. For performance testing the Test 
Administrator tryout is as important as the test subject tryout, if not more so. The purposes of the tryout 
are to ascertain whether the written instructions communicate as intended and to determine the feasibility 
of test procedures. In this project, several such tryouts were held at various stages of development and 
proved invaluable in pointing up where instructions were inadequate or confusing and for verifying test 
procedure feasibility. Such trials, of course, are much more meaningful if the samples of test subjects and 
administrators approximate the ultimate target population. 

Summary of Steps for JTPT Development 

1. Identify major classes of job tasks. 

2. Define and describe each class of tasks in terms of inputs, outputs, and processing conditions. 

3. : Select a representative sample of job products or processes to be measured, 

4. Construct a frame work of subtests referenced to the main criteria. 

5. Specify standards by which to evaluate output. 

6. Simulate input and output conJitions in job-like ways. 
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7. Develop the administrative arrangements for the tests. 

8. Try out and revise the tests. 

Relation to Previous Efforts 

Performance tests are not new in the Air Force. Many have been developed on an ad hoc basis for use 
in research concerning training. During and after World War II the Air Training Command used elaborate 
and comprehensive check rooms in their technical schools. Some of these operations still exist on a modest 
scale.. What is new is the systematic approach for developing JTPT, the results of which will be more 
meaningful in terms of demonstrated performance of typical job tasks. Important aspects of this approach 
include the following: 

1.. A more precise description or definition of a job criterion in terms of job products has been used 
consistently throughout the tests. This criterion of specified job products is used (criterion referenced) 
rather lhan stating that some percentage of the total number of necessary performance steps in the 
production process is conect (norm reference). This is the essential difference between a criterion refer- 
enced test and a norm referenced test. 

2., But beyond that, the present study is intended to describe systematic ways of dealing with the 
many types of activities that are contained within a single job.: This is done through a classification of 
maintenance activities and a framework of subtests referenced to a single criterion of producing field ready 
equipment. Hectronic and mechanical maintenance jobs have the characteristics of many types of tasks 
within one job. To treat them, the concept of criterion referenced testing had to be extended in some way. 
This is a report of one way in which it was done in a manner making such tests administratively feasible. 

3. Each type of job activity has been considered separately for the purpose of test development and 
scoring. A number of tests have been developed for each type of job activity. 

4. Since the factors being measured are so complex and diverse, the result of each test is reported 
separately. No attempt has been made to report test results in terms of a meaningless single score for an 
individual. 



SECTION II. TESTDEVELOPME^^^ 
Introduction 

As stated previously, the prime equipment selected for this project was the AN/APN-147 Doppler 
Navigational Radar and its associated AN/ASN-3S Computer. This equipment is standard on several opera- 
tional aircraft and its maintenance is fairly complex. Responsibility for its maintenance is assigned to 
personnel with Air Force Specialty Code (AFSC) 328X4, Avionics Inertial and Radar Navigation Systems 
Technician. This AFSC is also responsible for many other electronic systems, and an individual in this 
AFSC may or may not be exposed to all of these systems during his time as a technician. The intermediate 
level of maintenance was the primary concern of this project. Most organizational activities also are 
included in intermediate maintenance activities. As a' result, tests for organizational maintenance can be 
readily developed using the tests developed for this project as models. 

Throughout the development of all of these criterion referenced JTPT, the steps outlined in Section I 
were followed. The following discussions concentrate on the peculiar problems that were associated with 
the development of each type of test. 

Ttoubleshootii^ 

The objective of the troubleshooting process is to isolate the faulty component and to return the 
equipment to a field ready condition. The first step taken was to identify the range of equipment faults 
that could occur by analysis of the equipment. This was done by identifying from the equipment 
schematics all of the "functional elements" or "stages" of the system. Then a piece part from each stage 



was identified as a candidate for fault insertion. These parts were selected so as to provide a basis for 
development of both "within-stage" and "between stage" troubleshooting tests, since both types of 
troubleshooting are required at the intermediate level of maintenance. Figure 3 lists the functional 
elements, references the schematic from Technical Order (TO) 12P5-2APN147-2, and identifies the part 
selected for faulting, along with its identification from the TO. 

These tests are administered at a normal work station where all of their usual job aids, such as 
references, test equipment and hand tools, are available. The test subject is told the equipment is mal- 
functioning and is given several symptoms of the type that would normally be on the tag when the 
equipment comes into the shop from the aircraft. The test subject is instructed to isolate and identify the 
faulty component. He is told that he may use any procedures, references, or test equipment which he 
chooses. (None are suggested, recommended, or graded). To this point the test situation exactly duplicated 
the job situation. However, at this point there is a deviation to increase administrative feasibility, The test 
subject is instructed not to unsolder any parts in the equipment. He is told that any time he wants to 
replace a suspected component he is to remove the module containing that part and request a module from 
the TA which contains a replacement for the specified part. The TA will take the module and, if the 
replacement part requested by the test subject was in fact the faulty one inserted into the equipment, a 
good module is returned to the student. If the replacement part requested by the test subject is not the 
faulty one, the TA, after a standard delay, will return the module to the subject indicating that the 
suspected component has been replaced. 

After the exchange, the test subject has to replace the module and determine whether the fault has 
been found (checkout). When the test subject performs the checkout and is satisfied that he has found the 
trouble, he lists the faulty part on his answer sheet and turns it in. That completes the test. His answer sheet 
can then be graded on the basis of right or wrong on a measure that is tantamount to producing the final 

product.: 

To provide additional diagnostic information about each test subject's performance, the Test 
Administrator will record each part that the subject requested and the subject will indicate what test 
equipment he utilized. In this way, a complete record of the individual's troubleshooting strategy will be 
available.: This information on process may be of interest, but in terms of scoring, only the faulty part listed 
by the test subject on his answer sheet will be graded. The test subject identifies the faulty part or he 
doesn't (go, no-go). (A detailed description of all graded items is included in Section III.) 

While this "module exchange" technique represents a difference from the job situation, incorrect 
estimates of the problem still penalize the test subject by requiring him to remove a module, present it to 
the TA, describe the suspected malfunctioning part, install the replacement module, and check it out. This 
process will generally be equal to or slightly less than the time normally spent by the technician in 
unsoldering, testing, removing, acquiring a replacement from supply room, and installing a new component. 
While the activities are some what different from the actual troubleshooting activities, they do not promote 
or reward significantly a typical behavior on the part of the technician (e.g., random trial and error 
searching does not have either a higher payoff or lesser penalities than in the actual job situation). 

Time to Perform 

The amount of time required to correct the equipment and bring it to field ready status is a function 
of a number of factors. There is some theoretical minimum time required to perfonn each process. But like 
a "four minute mile," the absolute minimum time is never known for certain in practice. The theoretical 
minimum time required for accomplishing each process in the test could be estimated. For instance, a 
technician might be rehearsed on a procedure for a day or so and then timed with a stop watch. But there is 
a question of what this would mean in terms of job standards. In electronic and mechanical maintenance 
jobs, there are no exact time standards on the job. There are estimates of what is "reasonable." But in the 
field shop job incumbents are not usually held to strict accountability for performing within these times. 
(Time to perform on the flight line at times is more critical than in the shop). 

So for these tests there were approximations made of how much time each process would require.. 
This time was "tried out" in pretrials, rounded off to quarter hour increments and used. The final time 
limits on each test item represent an amount of time that a supervisor would "not complain about" if a job 



20 23 



o 
I 



O 

o 



< 
o 
o 2: 









vO 










o 






1 


•H 






00 












;3 






£ 


cr 










X 


•H 






•H 




•g 






>^ / 











<A to 



6^ 



t>0 



Di < ^ 



o 

CNJ 



CNJ 



CNJ 
CNJ 



•s 



o 

o vo 
I to 

> ^ 



CNJ 



vO O 

CNJ 



i-H IT) 

o 

J. ^ 



CNJ CNJ 

o 

> ^ 

















PL, 
















CI. 












O 




o 








O 






LO 


o 




o 


CNJ 


CNJ 


o 




i-H 


o 


U 




1 




1 


LO 


J 


VO 








> 




U 





to 



LO 



•8 




8 

1 

.s 

I 

a; 



4-» 



< 
3= 
O 

00 
o o 

1 VO 

> ^ 



VO 
to LO 

O CNJ 



5> 
o Z 

I CNJ 



to 

J- ^ 



o 
to o 
o 



o 
I 



CNJ CNJ 

I 00 



< o 

o vo 

I LO 

> ^ 



00 VO 
o • 

I LO 



O CNJ 

I 2: 



to o 



4-» 

•5 

CO 



o 



u 
cr 



P^^O 
P^tO 

I 

CO 00 



t>0 



U 



P^vO 

C CO 00 
a; 



;3 
cr 

a. 



o 





















/- N 




LO 




LO 




LO 




LO 




LO 




rH 


u 








u 






i-H 


o 


1 


o 


t 


o 


1 


o 


1 


o 


1 






4-> 








4-> 












ct5 








ct5 






























;3 








;3 


bO 








•H 




•H 




•H 




•H 




• H 


o 




:g 




o 








o 






>^ / 




>^ / 




>^ / 




>^ / 




>^ / 


















/■ — > 




to 








LO 




VO 
















^ — ' 




>^ — ' 




>^ — / 






bO 



00 o 











o 






o 






u 


to 


c 








o 


I-H 




00 




O 


<w 








«H 








I-H 


bO 




c 






I-H 


o 




PL. 




u 




^ / 



ERIC 



21 



24 



incumbent took that amount of time to perform that task on the job. It is a great administrative con- 
venience to have set amounts of time for eact item. But the time limits should be set so that test subjects 
failed because they didn*t know how to perform, not because they weren't fast enough. 

The question of using time as a measure of performance effectiveness could be argued.. That is, it 
could be argued that a man who performs in less time than another man is "better." Time to perform is an 
important norm referenced concept: to compare times for individuals in training recyiired to reach the "go" 
criteria on the various criterion referenced tests may become an important factor for the standardization of 
selection tests. But time is not germane to the criterion referenced aspects of criterion reference tests as 
defined here - that is, the product which the test subject generates. Generally, individuals are not rated 
precisely on their speed of performance of maintenance activities on the job. To repeat, the time limits set 
for each test item is a time that supervisors would accept as OK. 

Prompting 

No "promptings" are allowed in the test. The TA is under strict instructions not to help or prompt a 
test subject. There is no way to standardize these "prompts." One type of prompt may make the difference 
between accomplishing the product or not. Another may save the test subject a minute or so and still 
another "prompt" may interfere with the test subject. So it is of little value to count the number of 
prompts, as they do not have an equal effect on test subjects. It is true that a man might receive a 
"prompt" on the job, particularly if he asked for help. But in such a case the man is not actually 
performing the whole task on his own, he is receiving help. Each performance test is designed to ascertain 
whether the test subject can perform the task on his own, not how well the TA can perform the task. And 
finally, when TAs are allowed to give help, they have been known to virtually "take over" from the test 
subject and do the job themselves. This array of reasons against promptings makes it very clear that TA 
must be strictly prohibited from prompting test subjects. Experience from this project indicates that TAs 
can be prohibited from this kind of interference with the test, but it does require sterr> admonitions, 
sometimes multiple. 

This is not to say that a TA cannot derive diagnostic information from observing a test subject's 
performance. That is, even if he is prohibited from prompting the test subject, the TA can still try to 
diagnose why the test subject gets "hung up" at certain points. (It is the opinion of the authors that the 
diagnostic nature of the subtests is much more useful than the diagnoses of a TA made during a test. A 
formal record of inadequacies on subtests is felt to be more useful to a training operation than are the 
fleeting impressions and hypotheses of a TA. This is not to say that the TA can't develop some judgements 
based on test results.) 

Application of Rationale 

With this general'' testing rationale developed and with the equipment functional components defined, 
it was then possible to develop a series of troubleshooting tests that represent within and between stage 
troubleshooting for the AN/APN-147-AN/ASN-35 system. Several sample tests were prepared along with 
the necessary test administration instructions and both were tried out on a group of technicians. This 
administration demonstrated that the component exchange rationale was feasible and easily handled by 
both the student and the TA. The ground rules concerning time to perform and prompting also were 
applied to the development of the later criterion referenced JTPT. 

It was also discovered that test subjects tended to go first into an alignment/adjustment routine when 
presented with an inoperative set of equipment. Since these skills were to be tested separately and since, for 
purposes of the troubleshooting tests, no random adjustments of the equipment could be permitted, a 
specific instruction was added to each test stating that the test problem was not a function of a misalign- 
ment.. 

Alignment/ Adjustment/Calibration 

Equipment calibration is a major task of the electronic technician. This task is variously called 
"alignment" and/or "adjustment." An initial attempt was made to differentiate between these terms for 
purposes of test rationale development. Adjustment is generally taken to mean manipulating controls to 



bring a series of equipment parameters into alignment. However, since alignment checking also requires 
control manipulation, it was found that no meaningful and consistent difference exists between them for 
testing purposes. Therefore, the same testing rationale was appKed to both. The test categories "alignment" 
and "adjustment," then, reflect only the terms applied to these activities by the official TO. 

To discover the nature of the alignment task, (i.e., inputs, outputs, processing), the procedures 
described in the TO for this equipment were exercised and analyzed. In attempting to carryout the 
alignment and adjustment procedures for the AN/APN-147-AN/ASN-35 system, it was found that the TO 
for these procedures could not be followed in all cases. If a technician attempted to foUow some 
procedures, he would not be able to accomplish the task. This poses a problem for the test constructor 
because he has no way to correct the TO in a formal authorized way in the time avaMable for test 
construction. So he should avoid such problems by selecting other items for his test. But in some cases, like 
the present one, this was not possible. One particular procedure was a very common one for the check out 
of many equipment repairs, so it was included in the test battery.. 

In this case the local solution was accepted as correct for the purposes of the test. This meant that 
input conditions for the test were established in accordance with the local procedure. The special test jigs 
were included as part of the standard equipment provided the test subject. He could use them or not as he 
chose.. The criterion was correctly adjusted or aligned equipment regardless of what process the test subject 
used to achieve it.: 

It is necessary to have measurement criteria which can be applied consistently from installation to 
installation, and peison to person. Therefore, alignment items were selected that give constant values and 
can be set and unset by TA without going through complicated adjustment procedures themselves. This 
requirement for administrative simplicity eliminated some alignment items frcnn inclusion in the test. But 
the few excluded items turned out to be some of the rarely performed alignment procedures. This trade-off 
was considered acceptable to simplify the test administratively. 

The testing approach devdoped was administered as follows: The TA sets up the problem by 
misadjusting a specified parameter. TTie Test Administrator's Handbook (AFHRL-TR-74-57(II), Part 11] 
includes instructions to the TA regarding the specific contrd to adjust and provides him with pictures of 
precisely where each control is located. The test subject's instructions then tell him to check a specific 
alignment (e.g., "Lock-Check Alignment'* or "Frequency Modulator Adjustment") and to adjust it if 
required. He then must demonstrate to the TA that the parameter has been adjusted to within tolerance 
limits. He does this by obtaining the required value on an appropriate item of test equipment and having 
the TA check it.. This is the criterion output measured by the TA. The TA completes an evaluation sheet to 
indicate whether or not the test subject successfully (go, no-go) performed that alignment according to 
specifications. The test subject comj^etes a sheet of diagnostic information on which he indicates what 
equipment he utilized in performing the alignment. This information is part of the diagnosis of his 
performance process, but not part of his score. 

Whenever test subjects are required to make adjustments or alignments, some subjects are going to 
make "over" adjustments resulting in badly misaligned equipment. The TA should be able to quickly 
realign the equipment. Therefore, in addition to the graphics and value descriptions for measuring the 
criterion output, a series of plastic adjustment templates or guides were developed. With these key guides 
the TA can quickly bring a set back to within acceptable operating conditions for test purposes. These 
templates are unique to a given type of equipment, but such aids can be constructed for any equipment. It 
was found during the test that the aids would work properly on all equipments used in the test though it 
had been anticipated that they might be unique to a given set of equipment. (Such aids could be considered 
for use on the job, but that is beyond the scq)e of the present project.) 

When subjected to preliminary tryout, these procedures proved to discriminate very well between 
individuals who could and could not perform specified alighments. 

Tests on Use of Test Equi|>. .^nt 

An examination of Table 1 indicates that the use of test equipment is an important common 
subactivity of all product producing activities. What the table does not show is that there are generally two 



23 



ERIC 



categories cf test equipment - special test equipments that are specific to a certain prime equipment such 
as the AN/APN-147 and the AN/ASN-35 and general test equipments that may be used with the 
AN/APN-147 and the AN/ASN-35 as well as with many other prime equipments. At the intermediate level 
of maintenance for the AN/APN-147 and AN/ASN-35, as it is cunently structured, the following seven test 
equipments are required: 

1 Tektronics Oscilloscope, Model 545B 

2. Voltohmeter,PSM'6 

3. Transistor Checker, Model 1890M 

4. Electron Tube Test Set, Model TV-2 

5. Audio Oscillator, TS-382 

6. Signal Generator, AN/URM-25D 

7. Doppler Generator, CMA-546A 

Of these seven test equipments, only one, the Doppler Generator, CMA-546A, is specific to the 
AN/APN-147 and AN/ASN-35. The AN/ASN-35 and AN/APN-147 equipment has the characteristic that it 
must be stimulated by this speckd generator and have special procedures performed on it in order to obtain 
outputs from it. The others are six general items of test equipment which are used.for testing many items of 
prime equipment.. The developmental considerations for each test equipment subtest constructed for this 
effort are discussed below. 

Oscilloscope Test 

The oscilloscope has a wide variety of features and capabilities: more than are needed in most jobs, 
including the present one. A special input simulator (waveform generator) was constructed to serve in place 
of the prime equipment for this subtest. It required the same type of output measijrements as did the prime 
equipment. Therefore, test subjects used those features of the scope which the prime equipment required to 
obtain diagnostic information. 

This waveform generator was designed to produce a variety of waveforms on the scope. A schematic 
of this generator is shown in Figure 4. The components were then enclosed in a box and a dummy circuit 
(Figure 5) depicted or the cover of the box, integrating the leads from the circuit inside, This was done so 
that accomplishea technicians could not solve the problems by inspection of the simple circuit inside. 

A series of equipment outputs was then defined which the test subject has to obtain on the scope. 
Each of these outputs is the waveform present at a specified test point. He must then compare the 
waveform obtained on the scope with the one drawn on the answer sheet for that item and indicate 
whether the two are within specified tolerances of each other.. The waveform obtainable at aiiy given test 
point is, of course, constant. By varying the standard given on the answer sheet, however, the answers to 
different forms of the same problem can be varied. This 's a substantial ^n in administrative feasibility. 

The test subject also recorded certain contrcJ settings used in the problem. He must have properly 
adjusted the equipment and properly evaluated the value of the resultant waveform to get a "'go^' rather 
than a "no-go." To disguise the critical control settings, several extra settings are required to be recorded. 
These are not evaluated in the scoring. The test subject's answers, then, are recorded on paper and can be 
processed immediately or later. 

The test probe calibration is an operation which the TA must evaluate separately.. This is necessary to 
insure that the probe was calibrated before each test subject started his test. It was included as the first 
problem for administrative convenience. 

Several tryouts of this test were required to debug the instructtoris associated with the test. These 
instructions were more complicated than those required in other tests and it was also difficult to find 
personnel sufficiently familiar with the scope operation to apply the instructions. This provides evidence 
that training in the practical use of the scope should be improved and that the present subtest would be a 
useful one in electronic technician training programs. 




Voltohmmeter Test 

The primary skills associated with voltohmmeter usage are voltage range settings, meter reading, and 
knowledge of where to apply it on the equipment. The last named skUl was not included in this particular 
subtest'because it is part of each troubleshooting test. 

To provide a standard test environment, a voltage generator was constructed to give a variety of AC 
and DC outputs. The different voltage outputs and resistance values are achieved by turning the selector 
switch to specified positions. The test subject's answer sheet lists specified value tolerances for each 
position.. The test subject must measure the existing value at that position and indicate whether he found it 
to be in or out of tolerance. This permits, as in the oscilloscope test, the development of alternate test 
forms. A schematic of the AC/DC voltage generator is displayed in Figure 6. The test box in which it is 
enclosed is displayed in Figure 7. 

Transistor Checker Test 

The 1890M Transistor Checker is an in-circuit testing device that permits the testing of a transistor 
without removing it from the circuit. To provide a suitable subtest, a printed circuit board was utUized that 
included a variety of transistors. Since the 1890M provides a wide range of measures, the subject is required 
to make a series of measurements on several test transistors and to record both the parameter values found 
and whether the transistor is good or bad. This provides an evaluation of the test subject's abOity to employ 
all of the features of the 1890M. Alternate forms of the test were developed by varying the transistors to be 
tested. It might be noted that this subtest does not require the test subject to use this equipment in the 
tests. It merely measures his ability to use it. 

Electron Tube Test Set, Model TV-Z 

The TV-2 checks a number of parameters of vacuum tubes, as the 1890M does for transistors. 
However, the TV-2 checks tubes on an out-of-circuit basis. The test subject is required to record a series of 
tube parameter values for several specific tubes and also indicate whether the tube is good or bad. Alternate 
test forms are provided by varying the tubes provided. 

Audio Oscillator, TS-382 

The audio oscillator is a frequently used item of test equipment. It is used to generate standard signals 
for input into a system for alignment purposes. The technician must be able to use this piece of equipment 
to perform various adjustments on the AN/APN-147 system. 

To test the test subject capability to use the device, he is told to set up for a specified equipment 
alignment.. He is further instructed that he must utilize the TS-382 in this alignment process. This and the 
criterion measure are the only difference between this test and the corresponding alignment test. When he 
has the equipment set upand operating, the test subject must demonstrate to the TA that he is getting the 
proper value reading at a specified point on the prime equipment.: This indicates that the TS-382 has been 
properly connected and adjusted. The TA then completes an evaluation sheet to indicate whether the test 
subject was successful or unsuccessful 

Signal Generator, AN/URM-25D 

This equipment performs the same type of function as the audio oscillator except it generates signals 
in the RF range rather than in the audio range. The same testing approach is used as in the audio oscillator 
test, 

Doppler Generator, CMA-546A 

The doppler generator is used only with the AN/APN-147 and drives the equipment for a specific 
operational check. To test the subject's ability to employ this test equipment item, he is instructed to set it 
up and to demonstrate its use to the TA. The TA in turn has an evaluation sheet of standards to which the 
prime equipment must perform if the generator has been properly connected and adjusted. As the test 
f.ubject demonstrates the use of the generator, the TA completes the evaluation on each output specifica- 
tions. 
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Operational Checkout 



The equipment checkout task constitutes the beginning and ending point for all other task and job 
outputs. It consists of making a proper bench setup of the prime equipment and the doppler generator, and 
then exercising the equipment according to TO procedures to ascert^ that it is functioning properly. 

For test purposes the TA, after checking that the equipment is functioning properly, disconnects the 
bench setup. The test subject is then required to re-establish the checkout setup and demonstiate to the 
TA, via operational checks, that the equipment is functioning properly. The TA then records whether or 
not the test subject was able to perform the required demonstration. The TA scores the performance in a 
go, no-go manner depending on whether all the outputs are according to job specifications or not. In 
addition, there is considerable diagnostic information available which could be record'^d if desired. 

At the present time, it is felt that the Checkout and Doppler Generator subtests are too similar to be 
separate, and tlie Doppler Generator Test should be eliminated. Hie ability of the test subject to connect 
the wiring harness might be kept as a separate subtest, but a man could easily go through years of job duty 
without having to perform this task. 

Removal and Replacement Tests 

The technician must disassemble and assemble the prime equipment in order to carry out his trouble- 
shooting and alignment tasks and to replace faulty components when found. This subtest consists of 
identification of the proper component and proper use of hand tools so as not to damage the equipment or 
components. A series of typical components, that the test subject has to remove and replace on the 
AN/APN-147-AN/ASN-35 system, was identified. Each item constitutes a separate test in which he must: 

1 . Insure that the test equipment is functioning properly (perform operational check-out which will 
be already set up for him). 

2. Locate the component specified for removal. 

3. Remove the component from the equipment and show it to the TA. 

4. Replace the component in the equipment. 

5. : Deomonstrate proper component replacement by performing oiToperational chedc-out for the 

TA. 

The TA will evaluate and record the subject*s performance on these tests. The subject will be scored 
on the basis of whether he removed the rigjit component and whether the equipment functioned conectly 
upon re-assembly. To aid the TA, a series of photographs is provided showing the equipment and the 
component in both their assembled and disassembled configuratioiis. This test series does not include 
removal and replacement tasks that require soldering. Due to the high probabflity of equipment 
degradation, all soldering skills are evaluated separately in a subtest utilizing expendable components. 

Soldering Tests 

This series of tests was provided to cover the soldering tasks. The soldering task consists of several 
associated elements. Viewed in sequence, the total soldering task proceeds as follows: 

1 . Unsolder while disconnecting the proper leads to remove faulty component. 

2. Select proper replacement component.. 
* 3. Install replacement component.: 

4. Re-connect aD associated leads. 

5. Solder. 

The soldering tasks related to the AN/APN-147-AN/ASN-35 are of two types - conventional wire 
circuitry and circuit boards. To test both types, as well as soldering and associated tasks, two subtests were 
devised. One test requires modification of a conventional circuit and the other requires circuit board 
soldering. 
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For the conventional circuit soldering, a dummy circuit was buOt and installed in a box of moderate 
confinement. In this test the range of soldering tasks described above are evaluate d./Fhe TA must inspect 
the test subject*s work to insure that the proper component was removed, that the proper replacement was 
installed, that the replacement was installed with the correct polarity, that excessive heat was not generated 
on associated parts during soldering, and that a good solder joint resulted.: To perform this evaluation, the 
TA has a checklist, and the begirming and ending configuration schematics. The TA must treat the leads of 
the test circuit with heat sensitive material that liquifies if too much heat is applied in soldering. A 
schematic diagram of the starting configuration is shown in Figure 8, and a drawing of the enclosure is 
shown in Figure 9., 

The second subtest, is conducted on a circuit board.: The test subject is told what component to 
remove and which replacement to use. When he has completed the replacement, the TA inspects each 
solder-joint and compares it with pictorial representations of acceptable and unacceptable soldering. 
then records his evaluation of the soldering on an evaluation sheet. 

Summary 

The framework of subtests developed in this project, their purpose, and the approach used are 
summarized in Table 2. A samjJe of test materials including the test subjects information and identification 
sheet, test instructions and answer sheets are displayed in Appendix A. In addition, a complete copy o{ 
instructions for each test subject for all the JTPT developed together with the Test Administrator's 
handbook are provided under separate cover as AFHRL-TR-74-57(n), Part 11. 



SECTION DL ADMI^aSTRATIVE, SCORING AND DIAGNOSTIC CONSIDERATIONS 

Introduction 

Some of the administrative and scoring considerations have already been discussed in the preceding 
sections as they pertained to the test development technology and to the actual test development. The 
rationale of dependent maintenance tasks for electronic equipment maintenance, as well as, scoring 
considerations for these tests also have been discussed. But no scheme for systematically recording these 
scores and applying the rationale has been provided., A proposed profile for recording and analyzing test 
results in the light of the dependency rationale is presented in this section. 

Additional diagnostic information is provided by detailed descriptions of the scoring scheme 
associated with each individual test and a description of the diagnostic information provided by each test. 
Other considerations include individual and group uses of test as well as test security considerations. 

The Reporting of Test Results 

Tests, Problems and Scorable Results 

The scoring rationale used throughout these tests is that only specifications of job products are 
measured and each of these job products is scored on a "go, no-go" basis. Table 2 indicates that tests have 
been prepared for seven classes of tasks; namely, (1) checkout, (2) physical skUl tasks (soldering), (3) 
remove and replace, (4) test equipment, (5) adjustment, (6) alignment, and (7) troubleshooting. 

Table 3 summarizes the number of tests, problems, and scorable products by class of tests developed 
for the AN/APN-147 and AN/ASN-35. The simple addition of numbers shown in Table 3 indicates that 
there are 48 tests, 81 problems, and 133 scorable products. But these numbers tell us nothing in terms of 
the content of the tests. To say that one test subject accomplished 100 scorable products while another 
accomplished 90 tells us nothing about the job readiness of either of these individuals or that one is better 
than the other. The varieties of scorable products are so diverse that any combination of them without 
regard to what they represent is meaningless. The only meaningful presentation of such information must 
be in terms of a profile designed to attach meaning to such numbers. A sample of such a profile is shown in 
Figure 10. 
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ERIC 



32 



35 



Table 2. Major Test Classes 
• TEST CLASS PURPOSE 



APPROACH 



Alignment 
(AL Series) 



Adjustment 
(AD Series) 



Troubleshooting 
CTS Series) 



Test Equipment Usage 
(SE Series) 



Removal and Replacemen 
(RR Series) 



Peripheral 
(PT Series) 



Check-Out 
(CO Series) 



To determine whether a 
technician can perform 
a representative set 
of alignments on the 
AN/APN-147-AN/ASN-35 
system. 

To determine whether a 
technician can perform 
a representative series 
of adjustments on the 
AN/APN-147--AN/ASN-^35 * 
system. 

To determine whether a 
technician can locate 
and identify specific 
within-stage and be- 
tween-stage faults in 
the prime equipment. 

To determine whether a 
technician can proper- 
ly use the test equip- 
ment specific to his 
work assignment* 



To determine whether a 
technician can correct- 
ly identify, remove,^ 
and replace a prime 
equipment module and 
retain the set in an 
operational status. 

To determine whether a 
technician can solder 
properly under job 
conditions. 



To determine whether a^ 
technician can properly 
connect the subject 
prime equipment to its 
bench test set-up con- 
figuration and perform 
operational check-out 
of system. 



A specific alignment 
parameter is thrown out 
of tolerance and the 
technician must demon- 
strate that he can re- 
align the system. 

As specific adjustment 
is thrown out of toler- 
ance and the teclinician 
must demonstrate that he 
can readjust the equip- 
ment. 

A specific fault is in- 
serted into the equip- 
ment and the technician 
must locate and identify 
the faulty component. ^ 



The technician is re- 
quired to use a specific 
item of test equipment 
and demonstrate that he 
can properly utilize the 
features of that item. 

Modules will be specified 
for the test subject to 
remove and replace and 
upon completion," the 
operational status of the 
equipment will be checked. 



The technician must pro- 
perly identify, remove 
and replace a soldered 
piece part in a test 
circuit. 

Require the technician to 
connect the prime equip- 
ment to the bench test 
set-up 9nd demonstrate 
that the required systems 
checks can be made. 



34 



37 



Table i. Tests, Problems, and Scorable Products 
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Hgure 10, A profile for displaying the results obtained by an individual subject from a Battery of Job Task 
Perfortnance Tests concerning an Electronic System - the AN/APN-147 and the AN/ASN-35. The results for 
Q ' only two tests are indicated. 
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A Profile for Reporting Test Results 



The profile provides reporting information and space for each problem. For example: there are only 
two checkout tests; one for the AN/APN-147 and one for the AN/ASN-35. Each of these checkout tests 
represents a single maintenance problem. For each of these problems, there is only one output or product 
to be measured. When scored on a **go, no-go" basis, a test subject can receive only a I or 0 for either of 
these problems. This represents the case of one test, one problem, one scorable product. 

If a test subject took the two checkout tests and demonstrated that he could perform the checkout of 
the AN/ASN-35 (Test 2) but was unable to perform the checkout of the AN/APN-147 (Test I), these 
results would appear as follows on the profile chart. 



TESTS 



COj^ Checkout 




Possible score 



Earned score 



Bar presentation 



The "0" shown in line 2 for Test 1 indicates that the test subject was unable to accomplish Test 1 . The half 
filled cdl (K)) on line 3 under Test 1 indicates that the subject has tried Test 1 T)ut has failed. Hie "r 
shown on line 2 ^0T Test 2 indicates that the subject successfully accomplished Test 2. The filled cell (■) on 
line 3 presents this same information in bar form. If the subject had performed both checkout tasks 
successfully, the bar would of course be solid. 

The oscilloscope test (SE7) within the test equipment area represents another case. These are eight 
problems in this test. Most of these problems contain more than one scorable product. For example, 
problem 2 contains six scorable products. But the test subject must obtain all six scorable products or a 
score of "6" to receive a "go" on this problem. Since there are eight problems, the subject must receivd' 
eight "go's" to indicate proficiency in the use of the oscilloscope. As an example (next profile chart), a test 
subject was able to perform all the problems except problem 5 and on problem 5 he only obtained three of 
the seven products. 
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A glance at the bar ofsthe profile indicates that the subject has tried all problems of the oscilloscope test, 
and that he has successfully performed all but one of the problems. It, also, tcUs us that he is not fully 
qualified on the oscilloscope and will not be until he can perform the operations called for by problem S. 
The objective of course is full qualification. 

A nearly complete profile is shown in Figure 11. This profile indkates that the subject has 
demonstrated his ability to perform all of the test problems down to troubleshooting. To date, he has 
attempted nine of the eleven troubleshooting tests and successfully completed eight of the nine tests. He, 
therefore, is not completdy qualified in troubleshooting. 
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VigMTc U. A profile for displayit^ the results obtained by an individual subject from a battery of Job Task 
Performance Tests concerning an Electronic Svstcm - the AN/APN-147 and the AN/ASN-35.,This represents 
the profile of an individual who has successfully completed most of th|j^terv« 
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This profile is not presented as the final solution to the profile problem for JTPT for electronic 
maintenance. It does contain most of the important information regarding a test subject's success on the 
full range of tests. It gives a meaningful picture of the subject's job abflities as measured by the test battery, 
indicating the subject's strengths and weaknesses. An examination of the profile (Figure 1 1) indicates that 
most of the tests in this battery contain only one problem. For example, there are two checkout tests, 
having one problem each and there are eleven trouble^ooting tests having one problem each. There are two 
soldering tests; one has two problems and the other has three. The Voltohmmeter (VOM) test has 20 
problems. The subject receives no "credit" for a problem unless he obtains all of the expected products. No 
attempt is made, to combine these scoresin terms of meaningless numbers, 

\ Proposed uses of these tests and of this profile in field maintenance unit, formal training and research 
environments will be discussed in Section VI of this report.. The dependency column of the profile will be 
discussed later under diagnostics. 

JTPT as Analytical and Diagnostic Tools 

Diagnostics have come to have a rather negative connotation - thb identifying faults and 
ascertaining what has caused the faults. Criterion referenced JTPT if properi; ructured and used can 
provide a large amount of such diagnostics concerning what is wrong with the world of maintenance. They, 
also, can provide a great deal of information concerning what is right with the world of maintenance. (The 
word analytical has been added to the subtitle to reflect these positive aspects of the use of JTPT). The 
following paragraphs are concerned with ways and means for obtaining specifics about the world of 
maintenance. 

Job Readiness, There are a great many contention concerning what it takes and how long it takes a 
new enlistee to become "job ready," but there is little hard data concerning the empirical or job related 
validity of these contentions. In fact, we do not know a great deal about how truly "job ready" people with 
many years of maintenance experience are to say nothing about new people. A valid question follows - 
What will a battery of JTPT (such as those developed during this effort) tell us about an individual's "job 
readiness?" And the answer is a great deal if properly utilized - certainly a great deal more than scores 
obtained on a job knowledge test (such as the Specialty Knowledge Test) or scores obtained from tests on 
electronic theory. 

As discussed earlier, the test battery has been designed so that the pattern of successful and 
unsuccessful tert performance indicated by the profile gives a good indication of the job readiness of an 
individual.: The hierarchy of job activities and their dependencies are reflected graphically in the 
dependency column of the profile (Figures 10 and 11). A successful trouble isolation and repair activity. 
Tor example, is always dependent on a technician's ability to check out his equipment; it is almost always 
dependent on his ability to align and adjust; and it may be dependent on his abflity to remove and replace 
defective items including in some cases his ability to solder. If an alignment or adjustment activity is 
considered to be a maintenance activity in its own right, a similar set of dependencies becomes apparent. 
The successful accomplishment of such an activity is dependent on the maintenance man's ability to check 
out his equipment and to use tc.Si .equipment.: 

Returning to the troubleshooting activity, if a test subject is given a troubleshooting problem and is 
unable to find the trouble, the faQure could be a result of his inabflity to perform any one or a combination 
of the dependent activities discussed above. Without further information we can oiJy guess as to why. But 
if we apply the whole test battery and the proposed proffle to the diagnosis and find that he cannot use his 
test equipment well, we know that at least one pait of his troubleshooting weaknesses is his inability to use 
test equipment. If this weakness is corrected and he stfll has difficulty troubleshooting by the process of 
elimination, we can conclude that his troubleshooting strategy is faulty. 

From a further projection of this dependency rationale, we can conclude that a technician should 
demonstrate his ability to perform checkout procedures, to use his test equipment, to perform remove and 
replace procedures and to perform align and adjust procedures before he is permitted to perform trouble- 
shooting. In like manner, he should not be tested on his ability to troubleshoot until his test profile 
indicates that he is proficient in these dependent activities. 
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The test profile will provide the maintenance supervisor and manager and the training supervisor with 
an instrument which will tell them the degree to which an individual is job ready. The lowest step in job 
readiness is, of course, the subject's ability to perform his checkout procedures. The highest step in this 
structure of progression would of course be his ability to troubleshoot. 

In the use of such a profile, supervisors and managers should not conclude that because an 
individual's profile of tests taken a year or two in the past is still valid. The subject may have lost his 
proficiency in some important areas through lack of use-in the same manner, as a pilot loses his proficiency 
to operate an aircraft. More will be said about maintenance of proficiency later. 

In addition to the major diagnostic information to be obtained from the tests profile just described. 
Appendix B (Table Bl) provides a description in tabular form of additional diagnostic information available 
for each test in the battery.. This appendix also provides a short description of each test and a description of 
the product or products expected from each test These are important items of information for the 
description of mamtenance v* Jaknesses identified by the profile.: 

The Role of the Test Administrator (TA) 

The role of the TA was briefly discussed in Section II in conjunction with the rationale for developing 
the troubleshooting tests where it was indicated that the TA should not be permitted to prompt test 
subjects. In regard to diagnosis, it might be argued that the TA can diagnose individual's difficulties from 
watching his overall performance. The TA can be said to represent any number of roles. He can stand for 
the job supervisor who wants to know what quality product he is getting. He could stand for the training 
school that wants to know what kind of product they have turned out. He could stand for the research 
scientist who has developed a new method of documentation or training and wants to know how good it is. 
He also could stand for the analyst who is trying to determine why the test subject fails to perform 
perfectly. The TA can be said to represent all those roles. But in the present research, a model has been 
designed to take on all of these roles in an objective manner. In this model test, the TA exercises virtually 
no judgment except to say whether certain products produced during the test meet the product specific- 
ations. Other than that, he has the rde of a traffic cop-to see that test subjects move smoothiy through the 
testing process, without prompting. The framework of subtests and classes of test items available in the 
present test battery provides a much more objective type of diagnosis regardless of who is the TA. 

Job Coverage 

Although a great deal can be learned from the proper application of the test profile, it won't tell us 
everything; and the level of confidence is greater for some testing activities than for others. 

Checkout Procedures. The job coverage of checkout procedures is of course complete. There is one 
checkout procedure for the AN/APN-147 and another for the AN/ASN-35. If an individual performs these 
activities under test conditions, we can be confident th'at he can perform all checkout procedures. 

Use of Test Equipment. The coverage of the uses of test equipment for the AN/APN-147 and 
AN/ASN-35 also is complete. If a:* individual can generate the oroducts as called for by these test equip- 
ment tests, we can be confident that he can use his test equipment with the AN/APN-147 and AN/ASN-35^ 
The VOM and Oscilloscope tests have more general application. Even with their cunent coverage, we can be 
reasonably confident that an individual who can perform these test problems successfully can perform most 
of the capabilities of the VOM and the oscilloscope. With some expansion, all cat^uOilities could be 
included. The test concerning the transistor and tube checkipi; include all of the capabilities of these test 
equipments. 

Alignmenty Adjustment and Calibration. The sample of problems concerning alignment and adjust- 
ment is large and covers very typical activities. As a result, we can be Treasonably confident that subjects 
who perform these problems successfully can perform the aligmnents, adjustments and calibratioris required 
by the organizational and intermdediate levels of maintenance for the AN/APN-147 and the AN/ASN-35. 

Remove and Replace The sample of remove and replace problems is rather large and the problems are 
typical. In addition, most remove and replace actions are not too difficult. As a result, we can be reasonably 
confident that subjects who perform the ten remove and replace tests can perform all remove actions for 
the AN/APN.147 and the AN/ASN-35, 



Troubleshooting,, The testing of troubleshooting ability would seem to present a much greater 
sampling problem than the other testing areas just discussed. This would seem to be especially true when 
considering equipments such as the AN/APN-147 and the AN/ASN-35 which contain what is now 
considered older circuit technology. The AN/APN-147, for example, contains approximately 14000 Field 
Replaceable Units (FRU), each of which represents a potential fault to be isolated. Equipments which 
reflect more modern electronic technology have much fewer FRU. The Inertial Doppler Navigational 
Equipment (IDNE) on the C-5 aircraft, for example, contains only 28 FRU. The AN/APN-147 and the 
IDNE perform much the same operational functions. (The fewer FRU of course reduce the difficulty of 
test sampling as well as the difficulty of troubleshooting itself for the organizational and intermediate levels 
of maintenance.) 

When considered from the point of view of the number of possible troubles in the AN/APN-147 and 
the AN/ASN-35, a battery of only 11 troubleshooting tests is certainly a small sample. Considered as an 
isolated and independent test battery, such troubleshooting tests would certainly not tell us much about an 
individual's ability to troubleshoot, especially when we consider the many different human abilities or 
factors called for by the troubeshooting process. 

But when, considered in light of the entire battery of maintenance JTPT, we have a different 
sampling problem. If the test battery is administered as recommended, we will have previously determined 
that the test subject is proficient in aU of the important troubleshooting sub-activities except his cognitive 
activity of applying a troubleshooting scheme or logic or strategy (whichever we wish to call it). What we 
are primarily measuring under these conditions is the effectiveness of the test subject's troubleshooting 
strategy in a realistic job environment (which includes the normal interference of information gathering 
activities). The normal model of a troubleshooting strategy is the troubleshooting tree. There are fewer 
branches in any troubleshooting tree than the number of possible troubles. Another way of saying this is 
that each branch of a troubleshooting tree has many trouble signatures. The 11 troubleshooting tests 
require the test subject to samj^e all the main branches and many of the sub-branches of his trouble- 
shooting strategy. As a result, the 11 tests sample the branches of his troubleshooting strategy rather well. 
If the subject can solve these 1 1 troubleshooting problems, we can reasonably assume that he will be able to 
find most of the possible troubles in this equipment. But this rationale is only valid if he has previously 
demonstrated his sub-abilities on the other subtests of the battery before he is given the troubleshooting 
tests. Under these conditions the writers are of the opinion that the trouble-shooting sub-battery, becomes 
a powerful test instrument for determining whether or not the test subject can effectively troubleshoot the 
AN/APN.147 and AN/ASN-35, 



Innate Ability, Acquired Ability and Motivation 



When an individual performs a task in the test situation, it can usuaDy be considered his maximum 
effort at the time of the test. This effort will usually be somewhat greater than he would normally exert to 
perform the same task in a normal job situation. Such differences must be considered in interpreting the 
results of JTPT. 

An individual brings to a test situation or to his job situation at least three determinants-his innate 
abflity , his acquired abflity, and his motivation. Most individuals will make a special effort to increase their 
acquired ability prior to a test, if they know or think they know which tasks they are going to be required 
to perform. They will do this by practicing these tasks. In addition, they will be motivated to make a 
maximum effort during the test. For these two reasons, a test is normally a maximum effort situation. 

The JTPT such as developed for tliis effort can determine the test subject's ability (innate plus 
acquired) to perform the tasks of his job. Such tests will not predict how well the subject will be motivated 
to perform these tasks on the job. It, also, is difficult to predict how long an individual will retain his ability 
to perform tasks after he is tested. If he does not continue to practice their performance, he will lose his 
proficiency.: This rate of loss will vary from individual to individual, and will be greatly influenced by how 
well the performance of the task has been learned. It is, therefore, important that an individual **over learn" 
the performance of important tasks. It, also, follows that he should be tested frequently on important tasks 
to motivate him to practice enough to maintain a high level of proficiency.. 
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Croup vs Individual Measurement 



So far, our discussion concerning test administration and scoring has been orientated toward the 
measuring of the individual. It is sometimes desirable to know how well as group such as a maintenance unit 
or a graduating class can perform. The ideal way to ascertain such group information would be to give each 
member of the group the full battery of JTPT. But many times maintenance and training supervisors or 
administrators do not think thatsthey have the time nor the resources for such extensive testing. In such 
cases, a great deal can be learned about the group by administering the entire battery to representative 
individuals of the group or by administering selected tests of the battery to the entire group or to selected 
representatives of the group. The profile developed earlier can be modified to accommodate such admin- 
istrations. More will be said about these considerations in Section VI concerning suggested applications. 

Test Security 

Test security for "objective" type written tests has always been a problem. If a test subject is able to 
acquire a key to such a test (even if the key is not exact), it becomes rather easy for him to learn the 
necessary numbers, words, or word chains for obtaining a good score without knowing the subject matter 
the test is samplinig. With most of the subtests of this test battery, we do not have this problem. We are 
interested in the test subject performing typical tasks of his job. If he practices these tasks in preparation 
for a test, which he knows he issgoing to have, so much the better. His good performance on the test will 
result in better performance on the job. 

For example, the test subjectsknows he is going to have a performance test on checkout procedures. 
He knows that there is only one checkout procedure for each equipment. He knows that the checkout 
procedure is in the TO. He, also, should know that most TO checkout procedures are not entirely complete 
and should be practiced if he expects to receive a passing score. If he practices his checkout procedures 
before the tests, he will not only perform better on the test but he, also, will perform the checkout 
procedure better on the job. If he already can perform the checkout procedure, the added practice will 
contribute to desirable "over learning." This will contribute to his long range retention of the skill and to 
the better transfer of his checkout skill to unfamiliar equipment^. Much the same can be said for the 
remove and replace tests, the soldering tests, the align/adjust tests, and the test equipment tests. 

There could possibly be some compromise of the VOM tests if the exact voltage and resistance 
reading should happen to be known by test subjectsr-These values, however, can be modified very easily and 
should therefore be changed frequently. The sliort cut scoring scheme used in the oscilloscope tests could 
possibly result in minor compromise. But the oscilloscope tests are so complex that it is doubtful that such 
knowledge would help a test subject unless he already knows a great deal about using the oscilloscope.. 

The compromise of the "bugs" of troubleshooting tests could easily become a problem. The trouble- 
shooting tests have twc common characteristics with "objective" type written test items. There is only one 
right answer and the tests cover only a sample of the "subject matter." Precautions can be taken to 
overcome these weaknesses. The orders of administration should be different for every test subject. 
Equivalent tests can be produced for each test by faulting different signatures of the same subbranches of 
the troubleshooting tree. This would result in different faults giving the same or similar symptoms. Various 
combinations of equivalent tests could then be administered so that no two test subjects would receive the 
same combination of tests during any administration. 

If the test battery is administered as recommended, most of the abilities support troubleshooting will 
have been measured previously by performance tests which cannot be easily compromised. This would 
reduce the number of people talking the troubleshooting tests before having the necessary supporting 
maintenance skilly. The temptation to cheat on the troubleshooting tests would thus be greatly reduced. 



SECTION IV, TEST ADMINSTRATORS* MANUAL AND 
TRAINING PROGRAM FOR TEST ADMINISTRATORS 

The previous sections have discussed the JTPT development methodology; have described the 
development of a series of 48 JTPT in keeping with this methodology; and have discussed administr'^tive. 
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scoring and diagnostic considerations. No system of testing is complete until adequate instructions for test 
administration are available. In addition, the writers felt that a training program for TA, if not necessary, 
was highly desirable. This section is devoted to a description of the development of an administrator's 
manual as well as to a training program for TA. 

The Development of Instructions for The Administration 
of Job Task Performance Tests (JTPT) 

As discussed previously, JTPT were structured to facilitate administration as much as possible. What 
was now required were instructions on test administration procedures that were sufficiently detailed to 
permit administrative personnel, without extensive electronics backgrounds, to accurately administer the 
tests and evaluate performance. The basis for presuming that such personnel could be so utilized is the work 
which has been done for the Air Force on proceduralized job performance aids. 

Beginning with a number of AFHRL studies early in the 1960's, a set of techniques has been evolving 
for presenting complex job task instructions to relatively untrained personnel in such a manner that they 
could follow them and satisfactorily perform the tasks (Foley, 1973). Generally, the techniques involve 
breaking down the task into short, clear instructions, and giving an abundance of graphic support to the 
instructions. A sample of proceduralized instructions developed for the TA is included as Appendix C to 
this document. A complete set is provided as Part II of this Volume. The main elements of these 
instructions are: 

1. general information on how to organize and control the test administration environment; 

2. orientation to the AN/APN-147 radar and its associated AN/ASN-35 computer; 

3. specific step-by-step instructions on all aspects of each performance test, from set-up to 
evaluation and recovery 

General Administration Information and Equipment Orientation 

The general administration information covers such topics as overall support requirements, safety 
precautions required, test security considerations, and how to structure the testing environment. The 
orientation materials on the radar set and computer cover in detail the configuration, functioning and 
standard checkout procedures that pertain. The information in both of these sections is supplemented by 
the training program, discussed later. 

Specific Step-by-Step Instructions 

The instructions for administering each test are set up so that the TA can follow each step in 
sequence for complete test administration. The support materials and equipment required are specified first 
(both in text and graphics) and then the TA is told how to set up the test problem. In cases where problem 
set up requires maladjustment of the radar equipment or other complex problems, the test instructions 
provide graphic depictions of each step to support the written instructions. The same type of detail is 
provided for evaluating test subject performance. The results which are to be obtained are depicted in both 
texttial and graphic formats. 

An important part of test administration, especially if the TA are not qualified technicians, is 
recovering from test subject errors. When a test subject is unsure of himself, he may make some random 
attempts to solve the problem. This results in the equipment having additional malfunctions put into it 
besides the known test problem, or else the original problem is increased. As part of the testing package, the 
TA is provided a series of adjustment gauges which can be used to set critical parameters of the equipment 
without the use of test equipment. While this does not do the same caliber of job as would be required by 
normal maintenance standards, it does keep the equipment operational and permit testing to continue. 
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Not included as part of the original Test Administrator's Manual package are the evaluation and other 
administrative forms required. Due to the volume of materials, these were provided with the instructional 
materials for the test subjects as part of the original test development package. Both the Test Administor's 
Manual and the Test Subject's Instructions are now packaged in Part II of this volume. Neither the current 
administrator's manual nor the test package contains the scoring profile presented in Figures 10 and 11, 
Section III. This performance test project has been developmental. The profile was developed after the 
original JTPT and the administrator's manual had been developed and tried out. Such a profile should 
certainly be included as the reporting device for test results in any future test administration or develop- 
ment. 



The training program was developed, as were the test administration instructions, for individuals with 
little electronics background. It is geared to completely familiarizing the TA to the radar set and computer 
and consists ofi: 

1. Instruction on the mechanics of conducting JTPT, to include: scheduling, test security, 
simultaneous testing, and recording scores and other administrative data. 

2., Video tape of correction criterion performance of each of the JTPT on live equipment., 

3. Demonstration on actual equipment without power. 

4. Instruction on how to recover from equipment malfunctions induced by test subjects' errors. 

5. Practical exercises in test administration. 

Figure 12 shows the breakdown of the time allocated to the training program. 



The main thrust of the efforts reported in this volume has been concerned with the development of 
an administratively feasible battery of criterion referenced JTPT and adequate scoring schemes for the key 
tasks of electronic maintenance. The previous sections of this volume describe the test development 
methodology, the 48 JTPT developed, a scoring profile, the Test Administerator's Manual and a training 
program for Test Administrators. During the development of these tests, every effort was made to assure 
that each, test required the test subject to peiform a typical job task. As discussed in AFHRL'TR'74-57(I), 
these tests were designed to approach the ultimate job criterion as closely as possible. As a result, no 
validation of their empirical validity is possible. They are empirically valid by definition. 

The main objective of the tryout was the ascertain whether or not the testing system was 
administrable. During their initial development, each of the 48 tests was administered to at least two 
experienced technicians at McGuire Air Force Base, New Jersey. Some tests were administered three times 
at McGuire. The Administrator's Manual also, was checked out at McGuire. After necessary "debugging" of 
the Test Subject's Instructions and the Test Administrator's Manual, two members of the Military Airiift 
Command's (MAC) Maintenance Standarization Evaluation Team (MSET) were trained to administer the 
JTPT using the Test Administrator's Handbook.. During this training program at Scott Air Force Base, 
Illinois each MSET TA administered every test at least once. After being trained the MSET TA administered 
each JTPT during official field visits to other MAC bases at least three times. Each test in the battery 
therefore was administered at least seven times during jie development and tryout efforts described in this 
volume. Based upon these test administrations, and upon the performance of the MSET TA when they 
conducted the JTPT, it can be concluded that the battery of JTPT as constructed can be administered by 
Air Force Personnel and that meaningful results can be obtained from their administration. The main 
objective of ascertaining test administrability therefore was achieved. 
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SECTION V. TRYOUT OF JOB TASK PERFORMANCE TEST (JTPT) SYSTEM 



Purpose and Extent of Tryout 
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Original plans called for additional field administrations of the JTPT battery to obtain some needed 
hard data on how well experienced technicians can perform the tasks of their assigned jobs. Due to 
unfortunate circunstances explained in detail below. This desirable and important secondary objective was 
not achieved. In this regard tlie various JTPT administrations described previously to experienced 
technicians indicated that they could perform checkout and remove/icplare tasks. They did fairly well on 
soldering tasks. However they had trouble with the test equipment tests, the align/adjust tests and the 
troubleshooting tes*s. These JTPT were used as criterion tests for the validation of symbolic substitute 
tests. That effor* is described in AFHRL-TR-74-S7(IlI). In that effort eight troubleshooting JTPT were 
given to 15 ex; .rienced technicians. The data obtained in that effort also support the contention that 
experienced technician cannot trouble&hoot very effectively.. 

The comments that follow concern why MAC units were selected for the trial application described 
previously as well as some of the administrative difficulties experienced during these application efforts 
Some suggestions are made for future field applications in MAC. 

Selection of Trial Application 

As stated previously, the design of criterion referenced JTPT anticipated their use in research projects, 
in both formal and on the job training, and in field maintenance units. Therefore, realistic implementation 
of the performance testing system required that the tests be administered in operational units, by Air Force 
personnel, and in connection with a standard activity of the Air Force. The objective was to reduce as much 
as possible the "exceptional" nature of the testing activity generally associated with training programs or 
research projects, and integrate it into an existing activity to provide a more accurate estimate of the 
feasibflity of such testing. 

The support of the Military Airlift Command (MAC) was solicited, since they operate Maintenance 
Standardization Evaluation Teams (MSET). These teams are on a continuing program of evaluating 
maintenance activities of units within MAC, and they presently use a form of uncontrolled performance 
evaluation discussed below. This organization represented an area whose mission was thought to be highly 
compatible with the use of controlled performance testing, 

MAC authorities were briefed on the project, its goals and its methods, and they agreed to furnish the 
services of two of their senior electronics evaluators to conduct the JTPT during their regular evaluation 
trips on a "time-available" basis. 

Tryout of the Training Program 

The personnel who were assigned to the project were more highly qualified than had been 
anticipated. One NCO was fully acquainted with the AN/APN-147 radar and the other had an extensive 
electronics background. As a result both the test administration instructions and the training program were 
more detailed than they required. (Program objectives, however, required that the materials be developed 
for the more general type of test administrator originally anticipated, since it could not be assumed that 
such higfily-qualified personnel would always be available.) Due to the experience of the TA, coverage of 
this material was completed in about three days instead of five. After a description of the project and its 
goals, each of the JTPT was reviewed and a video-tape of its correct .performance shown. Test admin- 
istration aids and strategies related to each of the tests were then described and discussed. Practical work 
was given by taking the TA to a field maintenance shop and having them administer the tests to available 
technicians. 

As the final portion of the training program, the TA described their assigned duties during evaluation 
visits to MAC installations. Based on this, plans were made for test data collection and for the researchers to 
accompany the TA dunng their first subsequent evaluation visit. 

Field Tiyout of the JTPT 

Once the TA were trained, it was planned that they would administer as many JTPT as possible 
dunng their inspection visits over a period of several months. Test results would then be sent back to 
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project personnel who would evaluate them and set up a processing system that would begin converting the 
performance test results into meaningful analytical tools. 

At the outset of this phase of the program, two difflculties were encountered. The evaluation 
philosophy and practices of the MAC MSET program were fundamentally different from objective 
performance testing. The MSET operated on a policy of minimum interference with the operations of the 
unit being evaluated. They also covered a great deal of unit administrative procedures that were not the 
subject of the performance tests. In evaluating job or task performance, their approach is to observe 
technicians performing whatever work is in progress in the shop at the time. They evaluate a technician's 
performance in terms of use of technical documentation, safety precautions observed, and technical 
procedures fdlowed. While they may provide some incidental comments or evaluation pertaining to an 
individual, their primary mission is to rate the overall unit effectiveness according to common standards 
based on swft/ec/iVe judgements. 

This policy of not making any additional work conflicted with the requirements of performance 
testing, -sirtce the JTPT are objective measures of individual performance on standard work units under 
controlled condition The planned procedures for administrating performance tests require the diversion of 
personnel from their immediate tasks, even though they may be doing the same or simflar work at the time. 
These procedures insure that the input condition of the equipment must be known and constant, from one 
individual to another for each given test item. 

Since this MAC policy could not be waived during the evaluation period it was agreed that the TA 
would give such performance tests as there was time for them upon completion of the normal MSET 
evaluation work. As a result of this arrangement, only three batteries of JTPT were administered during the 
balance of the program. There were two reasons for this. First, there were fewer inspection visits conducted 
during the time period of the program than were anticipated. Whereas the normal schedule of MSET had 
been to conduct approximately one inspection a month, during this time only three such visits were 
conducted due to higher priority requirements. Second, there apparently was not sufficient extra time 
avaflable during the evaluation trips to permit the administration of the new performance test's. Most of the 
substantive equipment tests, such as troubleshooting or adjustment, require from one to two hours each to 
conduct. Unless such tests are part of the primary evaluation, such blocks of time are hard to come by . The 
tests also require a number of supporting elements such as faulted parts and standard testing devices to 
achieve the necessary consistency of test conditions. 

These support problems combined with the time required to administer such tests, are typical of 
administrative problems that surface when a new concept is attempted for U e first time. It is emphasized 
that all of these conditions are administrative hurdles. They in no way reduce the quality of the JTPT 
concept or the urgent require.nent for the use of such tests. Ways must be found to overcome such 
administrative problems. The remainder of this section and the next section of this report are devoted to 
this objective.. 

Benefits Expected from Policy Modification 

The following discussion is not intended to Reflect adversely on the present work of the MAC MSET. 
There is no doubt that the personnel of these teajns are doing a good job within the limits of their 
manpower and resources. The MSET evaluation is designed to insure that there are no major deficiencies 
and that the personnel appear to be competent in their jobs. There are, however, no strictly defined 
standards of performance or production in electronic repair shops today. The JTPT being tried out in this 
study provide a means for measuring performance and production according to strict standards. But a 
command decision would be needed to require the development and enforcement of strict standards. And 
the development of such standards for all avionics in MAC would require more resources than are presently 
available for the evaluative function. 

The spectrum of equipments found in MAC diops are but a part of the total spectrum of equipments 
assigned to personnel holding AFSC 328X4 Air Force wide. So the holder of this AFSC is actually a 
subspecialist in a MAC squadron. Any further specialization within a MAC shop makes the technician a 
sub-subspecialist. The criterion referenced JTPT being tried out in this study are concerned only with the 
AN/APN-147 Doppler Radar and its associated computer the AN/ASN-35. withm a MAC squadron 
specialization on one or another aspect of a Doppler Radar is not uncommon. 
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Although many holders of AFSC 328X4 function assub-subspecialists,each is officially supposed to 
be able to perform maintenarcj cn several equipments in MAC. If any of these 328X4's are transferred to 
other commands, they will be faced with different mixes of equipment. This raises a question of what 
standards of production a man holding AFSC 328X4 should be expected to meet. It raises still more basic 
questions in regard to the realism of personnel assignment policies. But, in the opinion of the writers we 
will never be able to improve maintenance Air Force wide until we have strict performance standards at the 
maintenance squadron level. The use of JTPT cannot resolve any of these policy questions but their use will 
certainly emphasize the requirement for finding workable answers to such questions. 

Assuming for the moment that strict standards of performance and production would be desirable, let 
us examine the implications. The JTPT would have to be broadened to include all equipments within a 
MAC squadron.. The periodic use of these tests would quickly identify the personnel who are "bottlenecks" 
in the production process. They would also provide practical diagnosis in what areas those personnel were 
weak; i.e., checkout, alignment, troubleshooting, and use of special or general test equipment. Tliis 
diagnosis would provide the basis for command action. It would provide specifics. And specifics are needed 
to accomplish effective corrective actions. Command action would be required to enforce correction of 
specifics, but it would not be the general type of action that says "shape up" without also indicating the 
specifics of how. 

Under present manpower and money restraints it may be impossible to develop JTPT for each of the 
large number of equipment specific tasks such as checkout, align, adjust, calibrate, remove and replace, and 
troubleshoot. But the limited number the JTPT on the use of general test equipments and on soldering have 
broad applications for most electronic technicians. In the opinion of the writers the development and 
enforcement of performance standards for the general test equipment spectrum of activities and for 
soldering are well within the current manpower limitations of a command like MAC. The enforcement 
aspect would, of course, require some change in MSET policy. A block of time for JTPT on the use of test 
equipments and on soldering would have to be scheduled for each team visit. In addition, a limited number 
of technicians would have to be diverted from their regular duties during each MSET visit in order to take 
these tests. 



SECTION VL APPLICATION CONSIDERATIONS AND SUGGESTIONS 

The Department of Defense as well as industry may be forced into using criterion referenced tests 
before adequate consideration has been given to their developmental technology and their effective 
introduction and utilization in the training and work environments. Serious consideration must be given to 
these matters or a great deal of money and time will be spent uselessly before effective criterion referenced 
testing systems become a reality. The previous sections of this report have presented a developmental 
technology for criterion refernced tests for electronic maintenance.. 

The purpose of this section is to suggest approaches for accomplishing the introduction and effective 
utilization of such tests. Although aimed primarily at criterion referenced JTPT for electronic maintenance, 
much of what is presented also will be applicable to the measurement of job success in other areas of the 
world of work.. The section begins with a statement of a number of objections to the use of JTPT, followed 
by a discussion of a number of topics which in the writers' opinion arc closely related to the introduction 
and utilization of such tests. This is followed by some specific suggestions for the introduction of such tests 
into the Air Force work environment and into training. Some suggestions, also, are made concerning future 
research and development considerations. 

Objections 

When compared to the cost of paper and pencil objective tests, criterion referenced JTPT arc more 
expensive to develop in terms of time, personnel and equipment both in the field and in training. And 
there is no doubt that JTPT do cost more to administer than paper and pencil tests, especially in the formal 
training situation even if they are administered by classroom or shop instructors. When the administration 
of such tests are assigned to the classroom or shop instructors, many times they do not find the time to 
administer them properly. When they are administered property by a special unit (as in Air Training 
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Command checkrooms) they are even more expensive. Such a unit usually requires additional personnel and 
equipment. Wlien administered by supervisors of a maintenance unit, less time is avaUable for these 
supervisors' other activities. 

A corollary to the first objection is that when job performance tests are given in {he formal training 
situation they emphasize the requirement for sufficient equipment in the shops or laboratory. The usual 
result is a requirement for more equipment. If students are to be tested on job tasks, they must have 
sufficient opportunity to practice these tasks. This added practice on school equipment usually results in a 
requirement for more maintenance personnel., [The writers do not believe this latter requireirent to be 
legitimate since the students and the instructors should be able to maintain their own equipment.] The 
added practice also causes the equipment to wear out faster. 

The administration of criterion referenced JTPT results in scores having no variance against which to 
standardize aptitude tests. If administered against' a go, no-go standard, criterion referenced tests by 
definition result in no continuum of test scores from subject to subject, much less a normal distribution of 
test scores. The result is two groups of subjects - one including those who can accomplish the task under 
test and the other including those who cannot. Such go, no-go scores cannot be handled by parametic 
statistical procedures traditionally used to standardize aptitude tests. 

Some test and measurement psychologists prefer to use single factor statistical procedures when 
standardizing aptitude tests. JTPT such as the ones developed for this project are not single factor tests. 
Such tests to be truly representative of the maintenance jobs must inclui^e many factors; and to be 
meaningful, their reported results must reflect many factors meaningfully. 

Application Considerations 

There are a number of factors concerning or closely related to JTPT that have been considered in the 
preparation of the suggestions and recommendations for the implementation and use of JTPT. These 
factors are discussed in the following paragraphs. The writers also believe that these paragraphs provide 
some necessary background information for anyone considering the recommendations of this document. 

Disregard for Empirical VaDdity 

It should be noted that most of the objections ignore the fact that paper and pencil objective tests are 
not empirically valid measures of job ability. The fact that they are given, the fact that great confidence is 
placed in their resuHs by most of their users and the fact that many important and expensive personnel 
decisions are based on their results, indicate that the validity of these such tests is seldom questioned. Many 
of the reasons given for using paper and pencil tests are based on administrative convenience totally 
disregarding empirical validity., Lyman (1971) has indicated that "Almost anything else can be forgiven if 
the test has very high (empirical) validity.*' Tne writers of this report are convinced that the current 
disregard of empirical validity in field and training situations would be unforgivable if the people involved 
really understood how invalid their current testing practices really are. No matter how cheaply paper and 
pencil job knowledge tests can be prepared or how easily they can be administered, such tests are not a 
bargain. Their results are often meaningless in terms of ability to perform maintenance tasks. AFHRL-TR- 
74-57(1) contains a detailed discussion of criteria and validity. 

The writers, also, are aware of the fact that many current personnel and training systems arc built 
around paper and pencil testing practice. To change such systems suddenly could be disastrous. What we 
recommend is an orderly modification of the current training and personnel systems. Wc purpose that JTPT 
be introduced so that the greatest improvement in maintenance efficiency can be obtained with the least 
disruption of our training and personnel procedures. But if Air Force is to have more efficient maintenance 
and formal training for maintenance, it is extremely important that the Air Force obtain hard data on how 
well maintenance men can actually perform the key tasks of their jobs. All available hard data would 
indicate that maintenance both in and out of the Air Force, although sometime effective, is very inefficient. 

Part of this inefficiency results from many Americans having undue faith in written tests. An example 
is cited from the civilian world of maintenance. Certification of auto mechanics has recently received 
considerable attention. Such certification is based on the ability of the test subjects to pass a scries of job 
knowledge tests on such subjects as engine repair, engine tune up and brake repair. What research tells us 
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about such tests would indicate that the empirical relationship between the mechanics' success on this series 
of tests and their ability to actually perform these job activities is not very high. Little reliance can be 
placed m certification of maintenance proficiency which is based on scores obtained from paper and pencil 
job knowledge tests. But some garages require all of their mechanics to be certified. 

Proficiency 

For efficient operation, the acquiring of and maintenance of proficiency to perform the key tasks of 
their jobs arc just as important for the maintenance personnel as it is for pilot personnel. As we all know, 
the Air Force as well as commercial airlines, have developed very extensive systems for insuring that their 
pilots have a high level of proficiency on the key job tasks before they are placed on-the-job, and for 
insunng the maintenance of their proficiency once they are on the job. Although the command 
maintenance standardisation teams are a small step in this direction, there are no comparable testing 
systems for maintenance men. The standardization teams in their present stage of development do not 
insure that each maintenance man has obtained a prescribed level of proficiency in the performance of the 
key tasks of his job. 

Just because a man works on a job dafly does not insure that he is maintaining his proficiency on 
important jo6^ activities. An example, in the case of the pilot is instrument flying. It is certainly important 
that a pilot have a higli level of proficiency in this area of hisjob. it is highly possible that he could fly 
for niontlis without having to rely solely on his instruments. As a result, when faced with a bad weather 
situation, he would not be able to perform, even though he had adequate proficiency at some time in the 
pjst In 1 real sense, the electronic maintenance man is "flying on instruments" most of the time; that is, he 
must use his t^'st equipment to determine what the electrons are doing since he cannot see them. Yet most 
maintenance men have never been required to take test equipment job performance tests.. No one really 
knows {induiJiiig themselves) how well or how poorly they can operate these instruments. Even if a 
technician had demonstrated at one time that he could use all the capabilities of his oscilloscope, there are 
some iniportajit capabilities of this instrument which he uses infrequently on his particular job. He usually 
cannot iraintain a high level of proficiency on this instrument unless he receives proficiency practice in its 
use Just as with the pilot's instruments, if the maintenance man was tested periodically on all the 
important uses of his oscilloscope, he would be motivated to maintain his proficiency. 

Proficiency and Visibility,, It would seem that those occupations or activities which are most visible 
are those in which a high level of demonstrated proficiency are demanded.: And where a high level of 
performance is demanded, formal performance tests in some form are given to insure an adequate level. For 
example, we all see people drive automobiles. Most states now administer "a behind the wheel" perform- 
ance test before issuing an individual a driver's license. Some states require a periodic retest after a person 
reaches a certain age. States and local governmental units have traffic control programs that monitor 
people's driving, which in a sense is a performance test on certain aspects of driving. As mentioned before, 
the Air Force and commercial airlines have very extensive job performance testing programs both in* 
training and after job assignment. These tests ascertain that the pilot has the necessary proficiency upon 
initial placement and to insure that he maintains such proficiency while assigned to the job. Air Force band 
members who constantly are on exhibit to the public, must pass a performance test on two instruments in 
order to qualify in their AFSC. Sports are another area in which people are judged on their performance. A 
professional football player is not hired or given a pay increase on the basis of his ability to pass a written 
examination on football. All of the mentioned occupations are highly visible. 

On the other hand, very few mechanics are ever required to demonstrate their ability to maintain an 
auto on a job performance test. And they repair most of the cars that are being operated by licensed 
drivers. As mentioned earlier, there is a national certification program, but this certification program for 
auto mechanics is based on a series of paper and pencil tests - not on demonstrated ability to actually 
perform maintenance tasks. And the public generally is not permitted to see the auto mechanic working. As 
far as Air Force maintenance specialists are concerned, they are qualified on paper and pencil job knowl- 
edge tests (Specialty Knowledge Tests). (Not like band members whose SKT is a perfoitnance test.) The 
civilian airlines have a much tighter system of certification. An aircraft engine or airframe mechanic must 
pass both a written and a performance test to be certified by the Federal Aviation Administration (FAA). 
But once he has his certificate, he is not given periodic performance tests to ascertain that he has 
maintained his proficiency. The FAA certification of electronic maintenance men is similar to the Air Force 
SKT where no performance tests are required.. 
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JTPT on the Use of Test Equipment. One of the expected benefits from the use of JTPT in the Air 

Force training and maintenance systems will be the improvement of maintenance. When implementing a 
technology it is always desirable to start with an area that effects the greatest improvement with the least 
expenditure of effort. For many reasons, performance tests concerning general test equipment would be an 
excellent vehicles for effecting such an improvement, 

1. The uses of such test equipments are key maintenance activities. An examination of the 
dependencies among maintenance tasks indicated in Table 1 and in the profi'.e presented in Figure 10 
indicates that the ability to perform most maintenance tasks requires the ability to use such test equip- 
ments. If a maintenance technician cannot use his test equipments well, he ce.lainly cannot perform 
dependent maintenance activities well. This especially is true of the oscilloscope and the voltohmmeter 
(VOM). For example, if an electronic maintenance technician cannot properly operate and interpret his 
oscilloscope or VOM, he can read troubles into his prime equipment that are not there, 

2. Such test equipments have general applicability across the entire range of electronic equipments 
and are used by all personnel possessing electronic maintenance AFSC&. Stated differently the use of test 
equipment is a '*g" factor common to all maintenance activity. As a result, the administration of a rather 
lunited number of JTPT concerning the use of test equipment, should, thus, result in a general improvement 
in the quality of maintenance.. 

3 . Based on available information few electronic technicians are expert in the use of the oscilloscope 
or the VOM. A comprehensive Navy study by Anderson (1962) covered 415 Navy Electronic Technicians 
from the Pacific Fleet. A smaller Air Force study covered thirty "5** and "7'* personnel (Foley, 1969). 
Both indicated that technicians were not able to use their equipment effectively. The results of these two 
studies are summarized from Foley (1969) in Table 4, 



Table 4. Percent of Correct Measurements for Each TVpe of Test Equipment 
in a Navy Study (Anderson, 1962) and an Air Force Study (Foley, 1969) 



Type of Test 
Equipment 


Number of 
Activities 


Percent of correct Meaiurements 
N»¥y N = 41 S» Air FO(ce N = 30** 


Multimeter (VOM) 


7 


73.5 


51.7 


Vacuum Tube Voltmeter (VTVM) 


4 


63.0 


76.6 


Signal Generator 


4 


57.1 


643 


Oscilloscope 


6 


32.0 





♦Sample included 68 chiefs, 55 first class petty officers (POs, 66 second class POs, 131 third class POs, and 95 

seamen. 



♦♦Sample included 5 and 7 level, mostly staff sergeants and technical sergeants. 



As mentioned earlier in this report, during the development and tryout of the test equipment JTPT 
for this project, a great amountsof difficulty was experienced when trying to find Air Force technicians 
who were expert in the uses of the 545B Tektronix Oscilloscope. Tliis is still more evidence supporting the 
contention that maintenance technicians generally are not proficient in using test equipments. Large scale 
administration of JTPT on the uses of common test equipment both in training and in the field would give 
us much needed, up-to-date hard data on how well electronic personnel can use their test equipments. 

4. Considering the large number of prime electronic systems now assigned to each electronic 
maintenance AFSC, it is impossible for a maintenance man to have an in-depth knowledge of each system. 

But if he can operate his test equipment with a high degree of proficiency and he has confidence in his 
ability , he can manage to perform a great deal of maintenance on an unfamfliar system. On the other hand, 
if he is unsure of himself on both his test equipment and his prime equipment, it is extremely difficult for 
him to function. So he is faced with either doing nothing or with removing and replacing components until 
he gets the system to operate.. 

5 . A common complaint of designers and managers of electronic systems Is that systems do not 
function with the sensitivity for which they were designed to operate. The writers believe that a major 
cause of this problem is that maintenance technicians are not able to properiy adjust and calibrate these 
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systems due to their limited ability to use test equipment, especiaiiy the osciMoscope. The writers also 
believe that there has been a great tendency on the part of many electronic maintenance technicians to 
avoid the use of the oscilloscope because they do not know how to use this instrument and as a result are 
afraid of it. 

Use of Part Task Trainers and SimulatoR. One of the reasons frequently sighted for not utilizing 

JTPT in training is the added equipment required for practice and testing. As soon as performance tests are 
given, the amount of **hands on" practice always increases. This is probably one of the reasons that such 
tests are not found in great numbers in Air Training Command. With only a limited number of equipmercS 
and a large number of students, the practice of checkout procedures alone soon wears out components on 
equipments which are not designed for such constant adjustment. There arc training innovations that can 
ease equipment wear and usage. Some of these .j' * * 

1. Photographic trainers for practicing ci..- ^t procedures, and for interpreting VOM, Vacuum 
Tube Voltmeter (VTVM), and oscilloscope displays. Unpublished work of AFHRL has indicated that 
subjects can learn check out procedures using full scale photographic trainers as well as they can using the 
actual equipments. 

2. Special inexnensively built signal sources that cover the range of capabilities of VOM, VTVM, and 
oscilloscopes. (Sa ipie ni such signal generators have been developed for this project.) Such signal sources 
make it possible ^< .udents and technicians to practice all of the capabilities of these tesi instruments 
without tying up prime equipments. 

3. Inexpensive part*task trainers ior practicing many of the maintenance activities. For example, 
alignment trainers could be constructed that would ^ ovide practice on many alignment behaviors. 

4. Full-task simulators for some jobs. Full task simulators can be developed for such tasks as the 
flight line checkout and troubleshooting to the blackbox level for many electronic subsystem found on 
aircraft. An example is found in the trainer developed for the Learned Centered Training Project (Rifkin, 
Keper, FoUey, & Valverde, 1969). 

In-Depth Training on One System 

Due to the large number of electronic systems assigned to most electronic maintenance AFSC, it has 
been considered impossible to give in-depth formal training on aU systems. The current formal training 
solution to this situation has been to expose all airmen entering all electronic AFSCs to a common 
electronic fundamentals course. This common training- is followed by "sets" training on several typical 
systems found in the specific AFSC to which the ainnen is assigned. The **sets" training is usually of the 
orientation variety in which the student receives very little **hands on" training. As a result, students, 
although exposed to several systems, receive no in-depth **hands on" training on any one system.. What 
"hands on" expenence each airman receives is after he is in place in his first job. The result is that he may 
never receive any in-depth training and practice in many key maintenance tasks even on that job. At the 
present time, it would be impossible to administer a complete battery of JTPT such as those developed for 
this project for any system dunng formal training because students have not been trained to perform many 
of the tasks covered by the JTPT battery. Many maintenance technicians will not learn to perform many 
niaintenance tasks with a high degree of proficiency unless they know they are going to have to 
demonstrate their proficiency on a formal performance test. 

Considering the current structure of the personnel system with regard to maintenance, the most 
effective but also most costly training system in terms of time, money and equipment, would be to provide 
in-depth "hanus on" trainmg supported by JTPT for every system on which a technician is required to 
perform maintenance. Considering the large number of systems in the Air Force inventory, such a solution 
is out of the question. But in the opinion of the writers indepth '*sets" training on one system would 
provide a solution that would insure i--oroved demonstrated maintenance performance during formal 
training. 

There are research results that would indicate that "hands on" in-depth training on one typical 
equipment is more effective than orientations on several system (Shriver, 1960; Shriver, Fink, & Trexler, 
1964). In this FORECAST Project, the students were given comprehensive JTPT on the one system on 
which they were trained to insure that they could perform all the maintenance tasks on that one system. It 
was found that these students could perform maintenance tasks successfully on similar equipments A^ith a 
lunited amount of orientation. Students, who had received orientation on a number of equipments, without 
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in-depth training on any one equipment, were not able to perform adequately on any equipment. These 
findings would mdicate that many complex maintenance behaviors, once learned well for one system, can 
be transferred to another system. In the opinion of the writers such transfer can be facilitated by training 
programs which are designed to teach transfer.. For example, maintenance procedures can be learned 
in-depth for one system foUowed by an orientation on a second system, which points out the maintenance 
similarities between the two systems. The technician wOl learn to look for such simUarities in unfamiliar 
systems. 

Individual vs Grorp Administration of JTPT 

This subject wa5 discussed eariier in this report with respect to the utilization of profiles. Further 
discussion is included here as a possible solution to the current application barriers outlined previously. In 
♦'i opinio*^ of the writerb, if management considers that the administration of JTPT on an individual basis 
I. unfeasible with respect to expense and time, their administration on a group basis, using systematic 
sampling techniques, should be given serious consideration. A great deal can be learned about the 
maintenance ability of a given maintenance unit or the relative effectiveness of abUities between units using 
such techniques. Technical training job effectiveness can be measured by applying the same type of sampling 
techniques to the members of each graduating class. As stated before, sampling could be across individuals, 
by giving selected representative individuals from each group the entire battery of tests. Sampling could be 
across classes of tasks; that is, give all members of one group a selected class of tests such as alignment and 
the next group tests concerning another class of tasks such as checkout, etc. Another mix could be to select 
one test such as the oscQloscope test and administer it to a limited number of subjects from each group for 
an extended period of time. This last suggestion would be ideal if it was decided to concentrate on use of 
test equipment to the exclusion of other performance tests. Many other sampHng mixes could be suggested. 
However, whatever sampling scheme is utilized there is no doubt, in the opinion of the writers, that 
valuable hard data would be obtained about quality of maintenance for manager and trainers. In addition, 
the quality of maintenance would improve in the areas of test coverage. 

Symbolic Substitute Test. A somewhat promising adjunct to actual JTPT are symbolic substitutes. 

Symbolic substitute tests are tests that rec uire the subject to perform in a manner simUar to the real world 
of maintenance, but not using aciual hardward. Such tests may use pictorial or inexpensive whole or 
part-task simulators. They are not traditional job knowledge tests. A follow on research project supported 
by AFHRL has developed some symbolic substitute tests., The development and limited tryout of these 
tests are reported in AFHRL-TR-74-57(in). They are mentioned here only because they have implications 
concerning the introduction of JTPT to both the Air Force training and field environments. For example, 
such tests if successful, might be used to supplement performance tests in troubleshooting. An individual 
subject might be given only one or two troubleshooting problems on actual equipment but a large number 
of such problems by symbolic substitution. However, siwh tests should not be used until it can be 
demonstrate * 'hat they have high empirical validity. 

Prediction of Training and Job Success. For over twenty years the Air Force has had a selection 

system that has included the use of prediction or aptitude tests. Personnel selected using these tests are 
successful when current training procedures are used. These aptitude tests have been standardized on the 
basis of the final graduating .ores of students. Such scores are ' -^avily weighted with the resuhs of paper 
and pencil objective tests scores. These end-of-course scores fori a very convenient basis for standardizing 
Aptitude since they usually produce a wide range of variance. 

Selection tests of this type have recently been subjected to a great deal of criticism because they have 
not selected personnel of different 'thnic groups equally. It has been demonstrated that many of the 
personnel who have been rejected by this system can be taught to successfully perform the tasks of the 
actual job when job oriented training procedures ar.* used. The measurement of training success under these 
conditions is based on the criterion of job success rather than on the critericn of paper and pencU tests. 

Job criterion referenced tests such as JTPT would provide a much more reahstic basis for 
standardizing aptitude or selection tests for training than the current type of school scores. However, go, 
no-go scores in themselves provide no variance for the current standardizing procedures. The writers believe 
that there are several possible solutions to this situation. One solution would be to use the time in training 
required for individual students to reach criterion as the norm for standardizing such aptitude tests. Under 
proper conditions, such time scores would probably provide scores with sufficient variance to apply current 
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standardizing procedures. More modern training technologies encourage individually paced training. If 
individual pacing *,vere applied in job oriented "sets" training, time to criterion should produce a sufficient 
amount of empirically valid variance., (This o|.course assumes that paper and pencil aptitude test items can 
be found that would discriminate on the balj^fif this source of variance.) The traditional "locked step" 
training practices, which* tend to force most ^^Rnts to complete training in a standard amount of time, 
would not produce dip nedSssary variance. ^^^^ 

Another possible solution would be to use^e number of attempts an individual student requires 
before he successfully reaches criterion on each JTFT that he is given during training. (This could be data 
gathered either during formal training or during on-tfie-job training.) Such a system would result in each 
class of job activity providing a separate factor again^ which to standardize aptitude test items. To use this 
approach, test psychologists may have to shift to iion -parametric statistical procedures to determine the 
predictivcncss of each aptitude test item in relation to the number of attempts on each JTPT. 

Suggested Applications 

Need for Hard Data Concerning the Performance of Maintenance 

The Air Force is authorized approximately 180,000 maintenance men. More than 80 percent of these 
airmen will spend only four years with the Air Force., Most of the first enlistment airmen receive rather 
extensive formal technical training as well as on-the-job training. As z result, the support of this 
maintenance personnel system is extremely costly.: The Air Force's large investment in initial training is 
soon lost.: Yet, there is no bank of hard data, even on a sampling basis, concerning how job effective and 
how job relevant this extensive training exercise is. And there is no bank of hard data on how well 
maintenance men can perform the tasks of their jobs. What limited hard data, which h^ve been obtained 
from time to time, would indicate that neither training nor job performance are too effective. But when 
such data arc presented the often heard reaction is "it can't be that bad." 

The writers have reviewed the limited hard data, and in addition, have observed maintenance in many 
field settings. They are of the considered opinion that **it is that bad" and that the Air Force maintenance 
establishment as well as many civilian maintenance establishments are getting very inefficient results from 
their maintenance training dollars. The only way we can determine how well individuals or units can 
perform key maintenance tasks is to administer well designed JTPT to airmen graduating from training and 
to airmen on the job. 

It is often said that the Air Force does not have the time nor the money for the development and 
administration of JTPT for all of the hardware systems in the Air Force. This may be true. But even a few 
well selected tests will give Air Force managers much better information than they now possess. In the 
following paragraphs some suggestions are made as to how the Air Force can start gathering such data on a 
rather modest basis for electronic maintenance, and at the same time develop better tools for measuring the 
true efficiency of training and maintenance. 

Suggested Uses of JTPT in Maintenance Squadrons 

Individual maintenance performance is a key factor in determining how efficiently the total 
maintenance of that squadron is performed., Maintenance can be performed effectively without being 
performed efficiently. Current evaluation procedures do tell us if maintenance is performed effectively; but 
as stated earlier, they do not provide specifics concerning the efficiency of individuils. Until such specifics 
are available, no effective corrective action can be taken to improve the overall effi;iency of a maintenance 
squadron. And no great improvement in the overall efficiency of maintenance in the Air Force can be 
obtained until more specific hard data are available based on the efficiency of individual squadron 
maintenance irsonnel. Until such hard data are available maintenance organizations cannot give specific 
feedback to their related training organizations as to their real training requirements. The Maintenance 
Standardization Team of each command is an existing organization that can be used for obtaining at least 
some of the necessary specific data. A gradual modification of their current procedures is suggested in the 
following paragraphs. 

JTPTc Use of Common Test Equipments. Since common electronic test equipment is used by all 
electronic technicians, serious consideration should be given to systematic administration of these JTPT 
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For the fiist few visits to each maintenance squadron, these tests could be administered to maintenance 
technicians identified by each squadron as proficient in the use of test equipment. Later such tests could be 
administered to technicians selected at random by the standardization team. The idea being that if the 
squadron supervisors expected such tests they would insure that all their technicians would be proficient in 
their use. The writers are of the opinion that the maintenance of a high proficiency by ail electronic 
technicians in the use of common test equipment would greatly improve the quality of maintenance. Such 
tests would also provide common measures to compare squadron effectiveness. 

Th^ Injection of JTPT on Prime Equipments into the Maintenance System. After JTPT on test 
equipment are implemented as suggested above, a second step would be to insert test items of equipment 
(from the JTPT) for repair in the regular work load of the unit. These items would be assigned for repair on 
a regular basis and would not be identified as special test items. Performance on these items would be 
measured by a TA according to the standard conditions of the JTPT. This action would be similar to that of 
driving a car with a known standard fault into a number of repair stations to obtain estimates of damage 
and cost to repair. The progress of a standard item through the unit could be tracked to determine the 
number of parts replaced, number and level of man-hours expended and quality of repairs made.: This 
would provide base line performance data with standard times by which performance across units could be 
compared and standardized. 

The third step woufd be to produce similar faulted units for other systems within the MAC squadron, 
and finally for all the equipments assigned to the AFSC. Th ^ items would serve as the standard reference 
for work in units. The way in which they were handled, on a routine basis, would serve as both a standard 
and as a diagnosis of what specific conective actions needed to be taken to bring units up to specific 
standards and keep them there. 

These steps represent a gradual implementation of a change in policy. Under the present policy, 
technicians are expected to "do their best," and the MSET inspections are used to determine that this is 
being done. The alternative provided by the criterion referenced JTPT is that men would be expected to 
meet a standard that is above their present "best." By improving performances, through diagnosis of 
specific weaknesses, the "best" performance of technicians would become better. 

Suggestions for Expanding the Use of JUT in Tfaining 

For the last five or six years there has been a movement in Air Training Command to increase the use 
of JTPT in technical training. Such usage in itself will help improve the job skills of students since 
instructors teach and students learn whatever is emphasized in the testing program. The expansion of this 
JTPT Program should be encouraged for this reason alone. However, the scores from such tests are 
combined in some manner with the many paper and pencil test scores administered during training. So 
when and airmen graduates, we really know nothing concerning his success in job like activities during 
training. 

Tests on Use of Test Equipment. As discussed earlier, standard tests on the use of common test 
equipments especiafly on the oscilloscope, the VOM, and the VTVM offer very effective vehicles for 
insurmg a high degree of proficiency in the common activities of all electronic maintenance jobs. One set of 
common tests could be used for all electronic maintenance students. Examples of such tests have already 
been developed for this effort. 

Students cannot be expected to pass these tests on a "go, no go" basis without sufficient "hands on" 
practice. Students cannot be expected to develop a high degree of proficiency in the use of their test 
equipments by watching someone else use them or by using them from time to time as an incidental aspect 
of their "sets" training. A common task oriented block of programmed instruction could be easily 
developed for this purpose. There is already a model for such a program for the Tektronix 545 Oscilloscope 
(Woods, Trudo, & Pieper, 1966). But more practice exercises would have to be added to that program. 

A profile of each student's success should be maintained and no student should be permitted to 
graduate until he has demonstrated that he can perform all of the required functions of these equipments. 
Such a test should be given to students upon completing the test equipment block of instruction. 
Immediately before graduation, these tests should be administered to at least a sample of students to 
ascertain how well they have maintained their proficiency. 

The use of these common JTPT would provide an opportunity for teaching many instructors how to 
administer JTPT in an objective manner.. 
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Tests on Checkout Procedures, Checkout procedures are important activities which performed for all 
electronic equipment. The writers suggest that each graduate should be able to demonstrate high 
proficiency in the checkout procedures on one or two typical equipments for his AFSC in both the 
organizational and intermediate maintenance ijnviornoments. This skfll would then be more easily 
transferred to other equipments in his AFSC. Since such procedures can be practiced on inexpensive full 
sized photographic mock-ups, JTPT on checkout procedures could be added to "sets" training without 
increasing wear on the actual equipment. 

Sugg^ons for Research and Development 

Development of a Model Job Oriented Training Program, Students given the current orientation type 
offsets" training would probably be unable to pass a battery of JTPT similar to those developed for this 
project. It is suggested that a model self-paced training program be developed for one broad based 
electronic AFSC. One typical electronic systems maintain by this AFSC should be selected for in-depth 
training and a batiery of JTPT should be developed for that system. The training program should provide 
sufficient practice on the performance of the typical tasks and should reflect the dependency reflected in 
the scoring profile in the ordering of such practice. 

This training should be followed by orientation training on a similar typical system, followed by the 
administration of a batteiy of performance tests. Successful performance would support the hypothesis 
that in-depth training on one system is transferred to similar equipments. 

Development of Aptitude Tests Based on Criterion Referenced Tests. Aptitude tests should be 
developed using the data available from the training and testing on common test equipment and from the 
suggested model course as the base for standardization. 

Reporting of Maintenance Research Results. In the past, a great deal of confusion has developed 
concerning the scope of coverage of various maintenance research projects concerning both training and 
measurement. A diagram, similar to that shown in Figure 1 , is recommended for inclusion in reports of such 
work so that the reader can easily ascertain the scope of the work. 

Needed Research and Development Required on Other Areas of Measurement No in-depth develop- 
mental projects of criterion referenced JTPT for typical mechanical hardware have been reported to date. A 
typical jet engine is recommended as the test bed for such a development. Work in this area is greatly 
needed if maintenance of engines is to become less expensive. 



SECTION m SUMMARY AND CONCLUSIONS 

This report describes the development and tryout of a Job Task Performance Test (JTPT) System for 
Bectronic Maintenance Technicians. In addition, a number of other factors related to the use of JIPT are 
discussed in the report. The summary and conclusion statements are listed under two appropriate sub- 
headings - Development and Tryout of JTPT and Other Relevant Factors Concerning the Development and 
Use of JTPT, 

Development and Tryout of JTPT 

1, The main thrust of this effort was on the development of the criterion referenced Job Task Perform- 
ance Tests. A comprehensive classification of tasks of electronic maintenance jobs already existed. 
Representative tasks were selected from each of these classes. The requirement to perform a task become a 
test of the test batteiy. A test was developed for each selected task. Every effort was made to insure that 
each test required the test subject to perform the task being tested in the same manner as he would on his 

ctual job, These developmental actions were taken to make these tests as near to the ultimate criterion, the 
job, as possible. On these bases the tests are assumed to be empirically valid by definition and no further 
proof of their validity is possible or necessary. It is realized that all aspects of job performance are not 
reflected in these tests. The ability to perform the job tasks required by the tests is an absolutely necessary 
aspect of the maintenance job, but it is not necessarily sufficient for complete job performance. 
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2. A total of 48 Job Task Performance Tests (JTPT) was developed covering typical organizational and 
intermediate maintenance activities for electronic systems. The activities covered include checkout; align, 
adjust and calibrate; remove and replace; troubleshooting; soldering; and use of test equipment. 

3. JTPT were developed for typical organizational and intermediate maintenance activities for electronic 
subsystems. JTPT were not developed for depot maintenance for electronic systems. No JTPT were 
developed for the maintenance of mechanical systems such as engines. 

4. The developed JTPT, together with the accompanying administrators' manual and training program 
for Air Force administrators, are intended as a model system for future developments of job task criterion 
referenced tests for electronic maintenance. A large number of the tests are system specific for the Doppler 
Radar, the AN/APN-147 and its computer AN/ASN-35. However, the soldering tests and the tests on use of 
general test equipment have applicability across systems. 

5. The criterion of success for any test problem is the generation of an acceptable product without 
assistance of another person. This is a go, no-go criterion. Process may be used for diagnostic purposes but 
no credit is given for accomplishing part of the problem correctly. If time is a critical factor for the 
particular task (such as checkout on the flight line), it will be included as part of the go, no-go criterion for 
that particular task. Otherwise, the subject is required to complete the problem in a reasonable time., 

6. The activities tested are not mutually exclusive. For example, the troubleshooting activity can 
int ude all the other activities. The activities can be arranged in a iour level hierachy of indusivencss as 
follows: (1) checkout, remove and replace, and soldering activities; (2) use of general and special test 
equipment; (3) align, adjust, and calibrate activities; (4) troubleshooting. 

7. This hierachy of dependencies among activities made possible the development of a diagnostic profile 
for reporting test results in terms of the "go, no-go" criterion (Figures 10 and 1 1). A further application of 
this hierarchy of dependencies requires that an individual demonstrate proficiency in first and second levels 
of the hierachy before he is permitted to be tested on the third and fourth levels. For example, a subject 
must demonstrate his ability to use general and special test equipment without enor before being allowed 
to take the align, adjust, or calibrate tests. He should not be permitted to take the troubleshooting tests 
until he has passed the align, adjust, and calibrate tests. This hierachy of maintenance activities also has 
similar implications concerning the order in which a newly assigned apprentice or technician should be 
permitted to perform the maintenance tasks on an unfamfliar equipment. In addition, it has implications as 
to the order in which JTPT should be introduced into the field and training situations. 

8.. No attempt should be made to combine the results indicated on this JTPT profile into a single 
numerical score - such a score would be meaningless in terms of how job successful or job ready an 
individual is. And certainly no attempt should be made to combine the residts obtained from JTPT with 
scores obtained from paper and pencil theory^ and fob knowledge tests. 

9. The go, no-go type of scores resulting from criterion referenced tests such as JTPT provides no 
variance from individual to individual, except the dichotomy made up of the group of individuals which 
achieves criterion and the group that docs not. Current procedures which standardize aptitude tests against 
a single final school score for each individual would also have to be modified. 

10. The JTPT were designed as model criterion referenced tests for use in field maintenance units; for use 
in formal and on-the-job training; and for use as criterion tests in research projects involving maintenance. 
Since field maintenance units are responsible for the performance of the activities covered by the UTPT, 
such units were deemed to be the most desirable situations for the initial tryout of the JTPT system. 
Another consideration was to reduce as much as possible the "exceptional" nature of the testing activity 
generally associated with training programs or research projects. It also was desired that the tests be 
administered by Air Force maintenance personnel. The Military Airlift Command (MAC) agreed that 
personnel of its Maintenance Standardization Evaluation Team (MSET)be trained as test administrators 
and that tests be administered in the MAC maintenance units during evaluation visits. 

11. Since the tests were developed to be empirically valid, the main objective of their tryout was to 
ascertain that the tests were administrable. A secondary objective was included for their tryout, that of 
obtaining a needed bank of hard data concerning how well a large sample of Air Force electronic 
technicians could perform key tasks of theirjobs. Only the first objective of the planned tryout was 
achieved. 



12. During and after development each of these JTPT was administered at least seven times to insure its 
administrative feasibility. In addition, the tryout of the JTPT system indicated that Air Force maintenance 
technicians can be successfully trained as administrators and they are able to administer tests to Air Force 
mamti nance personnel. * 

13. The secondary objective of the planned tryout was not achieved because it was not possible for the 
MAC Maintenance Standardization Team to administer as many of those tests as had been anticipated. 
Several factors contributed to this result. The standardization teams made fewer evaluation visits than 
originally planned during the tryout period. In addition, the team operates on a basis of minimum 
interference with the operations of the field maintenance unit being evaluated. Since this policy could not 
be waived during the tryout period, it was agreed that the team would give such performance tests as time 
permitted upon completion of their regular evaluation. As a result there was little time left for these formal 
performance tests. 

14. Currently there are no strictly defined standards of performance for the maintenance of electronic 
equipments.. /In J there seems to be no "felt need" for such standards at the present time., The current 
MSET evaluations are designed to insure that there are no major deficiencies and that personnel appear io 
be competent in their jobs. Until strictly defined standards are developed and enforced no great improve- 
ments in maintenance efficiency can be expected. Such a program would require several man-years of effort 
to develop and implcircnt, but some steps in this direction could be taken within present manpower 
limitations, such as, the administration of tlic t,-.*cral test equipinent JTPT. 



Other Relevant Factors Concerning the Development and Use of JTPT 

15. High empirical validity or criterion-related validity is the most important single attribute required of 
job-related tests. JTPT are criterion based tests and have high empirical validity by definition. All avaUable 
research studies indicate that most paper and pencil job knowledge tests and theory tests have extremely 
low empirical validity. But many Americans accept the validity of such paper and pencil tests without 
question. 

16. Paper and pencil job knowledge tests are more easily developed and administered than JTPT. They 
require no equipment. They usually require less time to administer. Bur they do not measure how well 
individuals can perform the tasks of the maintenance jobs', and maintenance men are paid to perform 
maintenance tasks. Such job knowledge tests are no bargain, no matter how cheaply they can be developed 
or how conveniently and easily they can be administered. 

17. Current personnel and training systems are built around paper and pencil testing practices. To change 
such systems suddenly could result in disaster. But an orderly modification of training and testing practice 
is possible. 

18. For efficient Air Force operation, the acquisition of and maintenance of proficiency for performing 
the key tasks of their jobs is just as important for maintenance personnel as it is for pilot personnel. The Air 
Force, as well as commercial airiines, have developed very extensive systems for insuring that their pilots 
have a high level of proficiency on the key tasks of their jobs before they are placed on the job, and for 
insuring the maintenance of such proficiency once they are on the job. Although the command standardiza- 
tion teams are a small step in this direction, there are no comparable testing systems for maintenance men. 

19. Soldering and the use of general test equipment (such as the oscilloscope, the Voltohmmeter (VOM), 
the Vacuum Tube Voltmeter (VTVM), tube tester, and transister tester) are common activities performed 
by all electronic maintenance personnel.. Available hard data indicates that many technicians do not have 
high proficiency in some of these activities. A common systematic training and testing program for insuring 
a high degree of proficiency in these activities would resu^* in great benefit to the Air Force. 

20. The administration of more JTH would increase the amount of **hands on" practice by students and 
technicians. Excessive practice on in-place prime equipments can cause wear. The use of inexpensive 
equipment substitutes such as photographic trainers, part-task trainers, and inexpensive simulators can 
greatly reduce the requirement for practice on prime equipments. 
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21 . Under the current Air Force personnel system, a person assigned to any electronic maintenance AFSC 
is responsible for the maintenance of a large variety of electronic systems. Initial training cannot possibly 
cover all such systems. However, in-depth "hands on" training, including performance testings on one 
typical electronic system, is probably more effective training for transfer than*'hands off orientations on 
many typical systems. 

22. The administration of the entire battery of JTPT such as developed for this project requires a great 
amount of time.. Under some circumstances, time does not permit complete testing. For situations such as 
end of training evaluations and evaluation of maintenance units, the systematic sampling across tasks and 
across individuals would provide valuable evaluative information - certainly much better than random 
unstructured observations. 



SECTION VIII. RECOMMENDATIONS 

1. There is very little relationship between success on paper and pencil job knowledge and theory tests, 
and ability to perform tasks of maintenance jobs. This information should be given wide publicity to Air 
Force people at all levels in maintenance, personnel, and training. The current confidence placed on the 
results of these paper and pencil tests is unjustified. 

2. JTPT on the use of common test equipments and on soldering should become a scheduled and 
reported part of each command standardization team's evaluation of a maintenance squadron. Eventually, 
subjects for such tests should be selected at random from the electronic technicians assigned to each 
squadron being evaluated., 

3. Each electromc technician should be required to demonstrate his proficiency in the use of general test 
equipment and in soldering on an annual basis, just as pilots must demonstrate their ability to fly on 
instruments. 

^. Rather than attempting off line administration of equipment specific JTPT, the command stand- 
ardization teams should inject "black boxes" with standardized faults into the normal equipment repair 
pipe line and observe the results. The diagnostic information obtained from such a modified administration 
of JTPT should be used for command action for improving personnel performance in specifically identified 
area, f weakness.. Repeated applications of equivalent tests should be used to insure that performance is 
brought up to the specified standard across all units responsible for this equipment. 

5, JTPT on the use of common test equipments and on soldering, together with appropriate programmed 
instruction packages requiring sufticient "hands on" practice, should be included in all /ormal Air Force 
electronic technician training programs. No student should be permitted to graduate from an electronic 
maintenance training program until he has successfully passed these JTPT; 

6. The hypothesis, that "in depth," "hands on'* training on one typical electronic system for an AFSC 
would be more effective for transfer of training than **hands off orientation on many such systems, should 
be tested. For this purpose, a model training program should be developed which would include the 
following characteristics: (1) it should be developed for one typical equipment or system of a typical 
electronic AFSC; (2) its criteria of success should be a battery of JTPT such as those described jn this 
report; (3) it should be self-paced and no one should be graduated until he successfully demonstrated 
criterion performance; (4) a profile such as the one described in this technical report should be used to 
report training success; (5) it should provide for a large number of "hands on" maintenance exercises; (6) 
the ordering of these training exercises should be based on the dependencies described in this report; and, 
(7) maximum use of inexpensive equipment substitutes; such as photographic trainers, part-task trainers, 
and inexpensive simulators, should be developed for and be used in this training program. 

7. After a training program with the desired charateristics has been developed, exploratory work should 
be started on an aptitude testing program, which would predict job success based on success in this training 
program and on JTPT rcsuhs from maintenance squadrons. The feasibility of such sources of variance as 
time to complete training and number of attempts required to reach criterion on each JTPT should be 
explored as well as the possible use of non-parametric statistical procedures. 

8, A model JTPT system should be developed and tried out for mechanical systems. A typical jet engine 
is recommended as the test bed for such a development. 
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APPENDIX A: A SAMPLE OF JOB TASK 
PERFORMANCE TEST (JTPT) MATERIALS 



!• Test Subject Inforaation Sheet 
2. Test Subject Identification Sheet 
3* Test SEy - Problem 5 - Oscilloscope Test 
Subject^s Instructions 
Subject *s Answer Sheet 
4. Test TS-1 - Troubleshooting Test - 1 
Subject^s Instructions 
Subject^s Answer Sheet 



1. The test materials for all JTPT developed for this effort 
provided in AFHRL-TR-74"57(II) , Part II. 
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TEST SUBJECT INFORMATION SHEET 

This series of tests is being conducted to determine job proficiency 
among electronics maintenance personnel. The tests cover the use of 
electronic test equipment « hand tool use, and equipment troubleshooting. 

Testing will be conducted in your normal work areas . You will be 
given specific instructions as to the order in which you will take the 
tests. 

This package contains all of the instructions and answer sheets for 
each test* As you are assigned to a test station » you will be told which 
set of instructions to use and how to proceed. 

Some of the instructions are rather complex and may need to be read 
several times to gain full comprehension of what is expected of you. It 
will be well worthwhile for you to take the time and read the instructions 
carefully. If there is any question in your mind as to what you are 
required to do» feel free to ask the test administrator. It is also 
important that before you actually begin the test you know what answers 
are required. 

You may direct requires for any technical data, test equipment and 
other support requirements to the test administrator. The test 
administrator will assist you with any problems that you may encounter* 

Please complete the Identification Sheet on the next page and turn 
it in when requested. 
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Identification No. 
IDENTIFICATION SHEET 



NAME 

^^ANK SOCIAL SECURITY NO. 

ORGANIZATION 



PRIMARY AFSC^ TIME HELD 



DUTY AFSC 



DOES YOUR PRESENT JOB INCLUDE MAINTENANCE OF THE AN/APN-147-AN/ASN-35 
SYSTEM? YES NO 

WHAT USAF SCHOOL COURSES HAVE YOU HAD? 

COURSE DATE LOCATION 
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TEST SE7 



PROBLEM 5, FORM 3 
PROCEDURES : 

a. Channel "A" on the oscilloscope will be used for this problem. 

b. Ground the oscilloscope to the "Ground" connection indicated on 
the Waveform Generator Console. 

c. Set the sweep to display seven pulse of the waveform present at 
Test Point 6710 on the Waveform Generator Console. 

d. Utilize the delayed pulse feature to display the center pulse 
(4th positive going pulse from the left) in the center of the 
graticule. 

e. Compare all of the resultant values of the waveform on the 
oscilloscope with the one shown below to determine if it is 
within tolerance of + 10%. 

f. Mark the appropriate answer box below to indicate whether the 
Test Point waveform is in or out of tolerance. 

g. Record your selected scope control settings on the Oscilloscope 
Control Setting Answer Sheet. 

h. Return your answer sheet to the Test Administrator to go to the 
next problem. 



"160 sec 



Within Tolerance Q 
Out of Tolerance 
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TEST SE7 



PROBLEM 5, FORM 3 

OSCILLOSCOPE CONTROL SETTING ANSWER SHEET 

CONTROL SETTING 

Trigger Mode (Time Base A) 

Horizontal Display 

DELAY Multiplier (Read-out) 

Stability (Time Base A) 

Time/CM (Time Base A) 

Time/CM (Time Base B) 



TE ST INSTRUCT I ONS 



ERIC 



TEST 

TS-l 
TIME ALLOTTED 

30 minutes 
INSTRUCTIONS 

a. A problem In the Radar Set, AN/APN-147, makes the system 
function Incorrectly. Troubleshoot and isolate the mal- 
function to the defective part. Use whatever techniques 
and equipment that are available to localize the problem.^ 

b. Radar inoperative - Malfunction is NOT an alignment or 
ad jus tment. 

c. DO NOT UNSOLDER ANY PART FROM THE CIRCUIT FOR ANY REASON . 

d. You can remove "plug-in" type parts such as tubes, crystals, 
relays, etc*, and test on appropriate test equipment. 

e. If you suspect that a certain resistor, capacitor, transis- 
tor or other non plug-in item is defective, remove the 
module containing that part from its chassis. Ask the Test 
Administrator for a replacement module in which that part is 
good. 

f. If you decide a tube or other plug-in is defective, ask the 
Test Administrator for a good replacement. 

g. After you receive a serviceable part or module from the Test 
Administrator, replace the item and recheck t^e system. 

h. Complete your answer sheet as soon as you have identified 
the defective part* 
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TEST TS-1 
ANSWER SHEET 

A. What defective part did you find? Name part and schemati'' designation* 



B. Check Test Equipment You Used for Troubleshooting 

I I Oscilloscope | [ Frequency Meter | | Tube Tester 

I t VOM I I Pofwer Meter { ] Transistor Checker 

I 1 VTVM □ Spectrum Analyzer I I Semiconductor 

Diode Tester 

I I Signal Generator | | Distortion Indicate r Audio Oscillator 

j I Sweep Generator 
List any Specialized Test Equipment You Used for Troubleshooting. 
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APPENDIX B: JOB TASK PERFOKMANCE TEST (JTPT) 
SCORING AND DIAGNOSTIC INFORMATION 
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APPENDIX C: A SAMPLE OF MATERIALS FROM TEST 
ADMINISTRATOR'S HANDBOOK FOR JOB PERFORMANCE TESTS (JTPT) 



1. Table of Content 

2. Test Administrator General Instructions 

3. Test SE^ (GE^) Tektronlc 545B Oscilloscope 

General Instructions 
Instructions - Problem 1 
Instructions - Problem 5 

4. Test TS~1 Troubleshooting TS-1 



1. The complete Test Administrator's Handbook Is provided In AFHRL- 
TR-74-57(II) , Part II. 
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TEST ADMINISTRATOR'S HANDBOOK 



TABLE OF CONTENTS 

Section A General Information 

Part I ~ Soldering 2 

Part II — Radar Set AN/APN-147 (V) 

A. Equipment Description 4 

B. Radar Set Control 6 

C. Drift Angle Indicator 8 

Part III — Radar Set Special Test Equipment 

A. Doppler Simulator CMA-544/T(H) ' 10 

B. Doppler Test Harness C>IA-543/T(H) 12 

C. Doppler Generator CMA-546/T(H) 14 

D. Doppler Spectrum Analyzer CMA-545/T(H) 14 

Part IV — Radar Set Bench Test Set-up 16 

Part V — Radar Set Checkout 36 

Part VI — Computer Set AN/ASN-SS 48 

A. Equipment Description 48 

B. Controls and Indicators 50 

C. Special Test Equipment - Computer Test Harness 

CMA-549/T(H) . .56 

Part VII — Computer Bench Test Set-up 58 

Part VIII — Computer Set Checkout 72 
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TEST ADMINISTRATOR'S HANDBOOK 



TABLE OF CONTENTS (Con't) 



Section B Test Administrator Instructor 79 

Peripheral Equipment Tests (PT) 80 

PT-1, Circuit Board Soldering 81 

PT-2, Electronic Plece/Part Removal and Replacement .... 86 

General Equipment Tests (GE) S5 

GE~1, 1890M Transistor Testor 95 

GE-2, Model TV-2 Electron Tube Test Set 100 

GE-3, Model 410B Electronic Voltmeter 106 

GE-4, Tektronlc 545B Oscilloscope 112 

Operational Checkout Tests (CO) 166 

C0~1, Radar Set AN/APN-147 Operational Check 166 

CO-2, Navigational Computer AN/ASN-35 Operational Check . . 170 

Special Equipment Tests (SE) \ 176 

SE~1, Signal Generator, AN/URM-25D Usage 177 

SE-2, Doppler Generator, CMA-546 Usage 182 

SE-3, Audio Oscillator, TS-382 Usage 188 

Removal and Replacement, Tests (RR) 196 

RR-1, Crystal, Y-6701, Removal and Replacement 196 

RR-2, RF Oscillator, V-6201, Removal and Replacement. ... 202 

RR-3, Frequency Mixer, CV-1186/APN-147, Removal and 

Replacement 208 

RR-4, Signal Comparator, CMA~213/APN, Removal 

and Replacement 214 

RR-5, Sequential Timer, TD-505A/APN-147 , Removal 

and Replacement 220 



87 8S 



TEST ADMINISTRATOR'S HANDBOOK 



TABLE OF CONTENTS (Con^t) 
RR-6, Crystal Mixer, CR-6702 , (Side A) Removal and 



Replacement 226 

RR-7, Blower Motor Assembly, B-501 Removal and 

Replacement 232 

RR-8, Track Resolver Drive Assembly, Removal and 

Replacement ' 238 

RR-9, Translator Drive Assembly, Removal and 

Replacement 244 

RR-10, Relay, D-410, ^.emoval and Replacement 250 

Adjustment Tests (AD) 256^ 

AD-1, Transmitter Output Power Adjustment 256 

AD-2, Modulator Adjustment 266 

AD-3, Crystal Oscillator V-6701A Adjustment 276 

AD-4, Lock-Check Resistor R-6806 Adjustment 286 

AD-5, Power Supply Adjustments 296 

AD-6, Potentiometer R-301 Adjustment 308 

Alignment Tests (AL) 318 

AL-1, Antenna S3mchro 318 

AL-2, 200KC Bandpass Filter 328 

AL-3, Groundspeed and Drift Circuits 338 

AL-4, Computer Binary Division Ratio 348 

AL-5, Computer Resolver Bridge Balancing 358 

AL-6, Zeroing Transmitter S)mchros 368 

AL-7, Zeroing Control Transformer 378 
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TEST ADMINISTRATOR'S HANDBOOK 



TABLE OF CONTENTS (Con't) 

AL-8, Zeroing Control Transformer CT-301 388 

AL-9, Zeroing Transmitter S3nichro TX-302 398 

AL-lO, Zeroing Transmitter Synchro, TX-301 408 

Troubleshooting Tests (TS) 418 

TS~1, Frequency Tracker Pofwer Supply 418 

TS-2, Modulator 425 

TS-3, I-F Amplifier 434 

TS-4, I-F Amplifier 442 

TS-5, Electronic Control Amplifier 45O 

TS-6, Electronic Control Amplifier 458 

TS-7, Signal Comparator 455 

TS-8, Signal Comparator 474 

TS-9, Frequency Mixer Stage 482 

TS-10, Sequential Timer Sub-Assembly 49O 

TS-11, Plug-In Board 493 
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TEST ADMINISTRATOR ItJSTRUCTIONS 



A. GENERAL INSTRUCTION^ 

1. Introduction 

The AN/APN-1A7-AN/ASN-35 test package contains a comprehensive 
series of tests developed to measure job performance of the elec- 
tronic technician. These tests encompass all phases of day-co-day 
preventive and corrective maintenance that technicians are respon- 
sible for in their respective repair activities. 

2. Test Administration Support Criteria 

For effective test administration, it is important that 
ordinary support materials, such as hand tools, test equipment, etc., 
are readily available for the technician. Adequacy in number and 
serviceability is of prime importance. Sufficient time should be 
allotted by the Test Administrator prior to the test schedule to 
check prime equipment, test equipment and other required support 
items for reliability. 

Test Administration locations will be at job sites at various 
installations. Each installation will vary in number of personnel 
to be tested; types of maintenance available; and in facilities 
available. All of these factors have to be considered before 
testing can be organized. 

Test Administrators should allow sufficient time at each new 
Administrative Center for familiarization and test support 
preparation. 

3. Safety 

Test Administrators must observe the technician to Insure that 
spfe maintenance practices are adhered to at all times. GROUND 
S. JETY considerations take precedence over all others. 

A. Recommended Technician - Test Administrator Ratio 

The number of technicians that can be tested at once will 
depend on three conditions : 

a. amount of test support equipment available 

b. number of technicians available 

c. experience of the Test Administrator 
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Items a and b are self-explanatory. Item c means that as the 
Test Administrator gains some experience giving the test, he will 
see ways of giving several technicians the same test at once and 
also ways of administering several different tests simultaneously. 

5. Test Security 

A number of features have been Vbuilt-in" to these tests to 
reduce the possibility of test compromise. For example, different 
forms of the same test are provided that may appear the same, but 
require a different answer. However, enforcement of test security 
is the responsibility of the Test Administrator and common pre- 
cautions should be exercised: 

a. Prohib't col labo nation between technicians during testing. 

b. Provide sufficient space between test stations for uninhi- 
bited individual work. 

c. Do not permit technicians not actually engaged in testing 
in the test station area. 

d. Insure that test instructions and all support items are 
accounted for at the conclusion of each test. 

e. Do not leave Test Administrator's Manual unattended. 

6. Organization of Test Admiaistration Materials 

Section A of this Manual provides general information on the 
AN/APN-i47 radar and AN/ASil-35 computer. It describes the equip- 
ment, the bench set-up for checking it out, and the procedures for 
conducting operational checks. 

The detailed instructions for administering each individual 
test are contained in Section B. This includes: test equipment 
required; test set-up procedures; technician evaluation standards; 
answer keys; and ways to recover from likely test situations. 

Bound separately in the accompanying manual are the Technicians* 
Test Instructions. This contains the supply of forms, instructions, 
and answer sheets that the technician will receive. It is important, 
however, that the technician never be given this whole set of 
instructions. The Test Administrator is to maintain control of 
these instructions as well as his own. Such materials as the 
technician being tested needs are to be removed and provided to him 
individual Ly. 
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The Technician's Test Instruction Manual contains: 



a. a general background description of the tests 

b. a background data sheet for the technician to complete 

c. individual test instructions 

d. alternate forms of test 

e. evaluation forms for each test 

The Test Administrator must select all the relevant materials 
from the manual, give them to tha technician and insure that he 
gets all the materials back at the conclusion of each test. 
Figure 1 gives an overview of the events that are to occur in test 
administration. (Note: Each technician tested will complete a 
background data sheet only once.) 

7. Evaluation Procedures 

Test results are obtained in two ways in these tests — the 
Test Administrator evaluates the technician's performance, and/or 
the technician records his findings. Answer Sheets are provided 
when the technician must record his findings. Performance 
Evaluation Sheets are provided when the Test Administrator is to 
evaluate performance. Supplies of both these forms, as required 
by each test, are provided in the Technician's Test Instruction 
Manual . 

8. Test Administration 

The PT series of tests (Peripheral Tests) can be set up in a 
separate area from the actual work location where the other tests 
will be administered. This area, however, should be relatively 
close and easily reached. The tests in the PT series can then be 
set up in a "country fair" fashion and technicians rotated through, 
them as they come from or go to the other test area (actual work 
location) . 

All of the other test series will be administered to techni- 
cians at their normal work stations. This is to insure that the 
technician hask available his normal supply of equipment, tools, 
references, et^ Under good conditions, where each technician 
has a set of piflme equipment, he should be given the test series 
in the followiri| order: GE, CO, SE, RR, AD, AL, TS, with the PT 
series taken as convenient. Wlien there is a shortage of equip-- 
ment, i will be necessary to utilize the available equipment as 
availability dictates. For example, if there was only one avail- 
able prime equipment test set, then only one series could be run 
at a time and test scheduling would have to be set up accordingly. 
In this case, test security has to be carefully attend to, since 
there is a greater possibility of exchange of information among 
technicians. Greater use of the alternate test forms would be 
required than under conditions of simultaneous testing. 
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9. Test Recoveiry Guides 



It is probable that situations will arise where technicians 
will unintentionally grossly misalign the prime equipment. A 
variety of guides are included to serve as short-cuts for the 
Test Administrator to return the equipment to an operational 
status. While it is important to get the equipment functioning 
at the best level possible, it was not feasible to gear the 
creation of these guides towards absolute peak performance. Such 
items as the trimpot settings and the template for coils, as they 
appear in the guides, are relative to the different equipment 
systems being used. Thus, it is anticipated that although the 
guides will not enable the Test Administrator toll^'perf ectly" align 
the equipment, they will enable him to render the equipment 
"operational." 
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TEST ADMINISTRATOR INSTRUCT IONS 



TEST 

GE-4, Tektronlc 545B Oscilloscope 
TIME ALLOTTED 

60 minutes 
SUPPORT MATERIALS REQUIRED 

a. Tektronlc 545B Oscilloscope with CA plug-In (1) 

b. Two 10:1 Probes (2) 

c. Instruction manuals for oscilloscope and plug-In 

d. Waveform Generator Console (3) (special test box) 

PRE-TEST SET-UP 

a. Insure that the above listed materials are available 
at each of the testing stations to- be used. 

b. There are three versions of this test - Forms 1, 2, & 3. 
Each contains eight problems. 
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c. If several technicians are being tested at once^ give 
different forms of the test to adjacent technicians. 

d. If technicians are being tested one after the other, give 
different forms each administration. 

e. Decallbrate the test probe by loosening the collar 
(see opposite page) and turning the sleeve 1/2 turn 
counter-clockwise. Retlghten the collar. 

TEST ADMINISTRATION PROCEDURES 

a. Have the technician read his Instructions for test GE-A. 

b. Note the time and Instruct the technician to begin the test. 
PERFORMANCE EVALUATION PROCEDURES * 

a. Problem 1 . 

1. This problem consists of calibrating the test probe. 
It Is the same on all forms of the test. 
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Figure C3. Scope Probe. 
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When the technicians has calibrated tht probe, collect 
his Instruction Sheet for Problem 1. 

Compare the Indication that the technician has obtained 
on the oscilloscope with that shown on the Answer 
Key (see opposite page). 

If the probe has been properly calibrated: 

Check "YES" on the Performance Evaluation portion 
of the Instruction Sheet 

Instruct the technician to continue the test. 

If the probe has not been properly calibrated: 

Check "NO" on the Performance Evaluation portion 
of the Instruction Sheet 

Assist the technician to properly calibrate the probe 
Instruct the technician to continue the test. 
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F/gure C4. Typical scope pattern answer presentation. 
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b. Problems 2 through 8 

1. When the technician has completed the problems, 
collect his Instructions and Answer Sheets and insure 
that his identification number is on them. 

2. Compare his answers to the Answer Key for that form 
(see pages 119 to 16S). 

3. Each problem has two parts: 

J) 

(a) Is the signal withir tolerance? 

(b) What control settings were used? 

4. Part (a) is to be answered at the bottom of the 
Instruction Sheet. If it is answered incorrectly, 
mark it with an "X". 

5. Part (b) is answered on a second sheet. Check the 
settings used to see that they are the same as those 
on the Answer Key. 

6. Where the Answer Key indicates "N/A", disregard the 
technicians settings. These are irrelevant settings. 

7. Mark any other answers that are wrong with an "X": 
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TEST ADMINISTRATOR ANSWER KEY 1 



GE-4, PROBLEM 5, FORM 1 
PROCEDURES: 

a. Channel "A" on the oscilloscope will be used for this problem. 

b. Ground the oscilloscope to the "Ground" connection Indicated on 
the Waveform Generator Console. 

c. Set the sweep to display seven pulses of the. waveform present at 
Test Point 6710 on the Waveform Generator Console. 

d. Utilize the delayed pulse feature to display the center pulse 
(4th positive going pulse from the left) In the center of the 
graticule. 

e. Compare all of the resultant values of the waveform on the 
oscilloscope with the one shown below to determine If It Is 
within tolerance of + 10%. 

f . Mark the appropriate answer box below to Indicate whether the 
Test Point waveform Is In or out of tolerance. 

g. Record your selected scope control settings on the Oscilloscope 
Control Setting Answer Sheet. 

h. Return your answer sheet to the Test Administrator and go to the 
next problem. 
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Within Tolerance ^ 

Out of Tolerance 
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TEST ADMINISTRATOR ANSWER KEY 2 



GE-4, PROBLEM 5, FORM 1 

OSCILLOSCOPE CONTROL SETTING 



CONTROL 

Trigger Mode (Time Base A) 
Horizontal Display 
DELAY Multiplier (Read-out) 
Stability (Time Base A) 
Time/CM (Time Base A) 
Time/CM (Time Base B) 



SETTING 

Any setting except Auto 
"A" Dly'd by "B" 
Approximately 4:60 
Fully Clockwise 
10 usee 
• 1 msec 
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TEST ADMINISTRATOR INSTRUCTIONS 



TEST 

TS-1, Frequency Tracker Power Supply Troubleshooting 

TIME ALLOTTED 
60 minutes 

SUPPORT MATERIALS ilEQUIRED 

a. Bench test set-up as described in Section A, Part IV. 
Test Equipment, to be available upon request. (See 
opposite page.) 

c. Hand Tools 

d. Defective 6080WA Tube 



PRE-TES SET-UP 

a. Perform an operational checkout of the Radar set to 
insure that it Is functioning properly. Use the pro- 
cedures outlined In Section A, Part V» 
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Figure C5. Equipment required for bench checkout of AN/APN-147. 
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b. Remove operative V-7207 (tube 6080WA) from Frequency 
Tracker Power Supply (see opposite page). 

c. Install the defective 6080WA tube in V-7207. 

d. Review the technician's Test Instructions to familiarize 
youfself with them. 

TEST ADMINISTRATION PROCEDURES 

a. Have the technician read his Instructions and put his 
identification number on his Answer Sheet and Performance 
Evaluation Sheet. 

b. Collect the technician's Performance Evaluation Sheet. 

c. Note tthe time and have the technician begin the test. 

d. When the technician wants to replace a suspected faulty 
part or module, he must remove It from the set (WITHOUT 
UNSOLDERING ANY COMPONENTS) and bring It to you. 
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When the technician requests a replacement component, he 
must specify the exact piece/part that he wants replac 

If the technician has not properly specified the part he 
\(ants replaced on the module he has turned in, make him 
re-specify. 

Once you are satisfied with the request, you either: 

1. Go to a remote area and exchange the module or part 
turned in for the good one, if the technician has 
identified the faulty part; 

OR 

2. Go to a remote area and simulate the exchanging of 
one module or part for another, if the technician has 
not identified the faulty component. 

Return the module pr part to the technician and inform 
him that the part he requested is now good. 
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i. Mark the part requested In sequence (1, 2, 3, etc.) on the 
technician's Performance Evaluation Sheet. 

j. When the technician has completed the test or time has 
elapsed, collect his Answer Sheet. 

PERFORMNCE EVALUATION PROCEDURE 

a. When the technician turns In his answer sheet, compare 
his answer with the correct one. 

b. Mark the technician's ans' ^r with an "X" if it is wrong. 

c. Fasten the technician's Answer Sheet and Performance 
Evaluation Sheet together, insuring that his identifica- 
tion number is on both. 

POST-TEST RECOVERY 

a. If the technician has found and corrected the malfunction 
Inserted into the equipment, perform an operational check- 
out to insure proper equipment functioning. 
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If the technician was ur^able to repair the radar set, 
correct the fault by re- installing the good tube (see 
page 621). 

Perform an operational checkout. 

If equipment is still malfunctioning, request assistance 
from local support. 
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